{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Interactive Polynomial Solution Space\n",
    "\n",
    "#### Michael J. Pyrcz, Professor, The University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a simple workflow, demonstration of the solution space of linear regression for machine learning-based predictions. In the future I will add general polynomial modeling. Here's some basic details about linear regression. \n",
    "\n",
    "* I have a lecture on [Linear Regression](https://youtu.be/0fzbyhWiP84) as part of my [Machine Learning](https://www.youtube.com/playlist?list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf) course. Note, all recorded lectures, interactive and well-documented workflow demononstrations are available on my GitHub repository [GeostatsGuy's Python Numerical Demos](https://github.com/GeostatsGuy/PythonNumericalDemos). \n",
    "\n",
    "#### Linear Regression\n",
    "\n",
    "Linear regression for prediction.  Here are some key aspects of linear regression:\n",
    "\n",
    "**Parametric Model**\n",
    "\n",
    "* the fit model is a simple weighted linear additive model based on all the available features, $x_1,\\ldots,x_m$.\n",
    "\n",
    "* the general form of the multilinear regression model takes the form of $y = \\sum_{\\alpha = 1}^m b_{\\alpha} X_{\\alpha} + b_0$\n",
    "\n",
    "* the specific form of the linear regression model takes the form $y = b_1 x + b_0$\n",
    "\n",
    "**Least Squares**\n",
    "\n",
    "* least squares optimization is applied to select the model parameters, $b_1,\\ldots,b_m,b_0$ \n",
    "\n",
    "* we minize the error over the trainind data $\\sum_{i=1}^n (y_i - (\\sum_{\\alpha = 1}^m b_{\\alpha} x_{\\alpha} + b_0))^2$\n",
    "\n",
    "* this could be simplified as the sum of square error over the training data, $\\sum_{i=1}^n (\\Delta y_i)^2$\n",
    "\n",
    "**Assumptions**\n",
    "\n",
    "* **Error-free** - predictor variables are error free, not random variables \n",
    "* **Linearity** - response is linear combination of feature(s)\n",
    "* **Constant Variance** - error in response is constant over predictor(s) value\n",
    "* **Independence of Error** - error in response are uncorrelated with each other\n",
    "* **No multicollinearity** - none of the features are redundant with other features \n",
    "\n",
    "#### Other Resources\n",
    "\n",
    "This is a tutorial / demonstration of **Linear Regression**.  In $Python$, the $SciPy$ package, specifically the $Stats$ functions (https://docs.scipy.org/doc/scipy/reference/stats.html) provide excellent tools for efficient use of statistics.  \n",
    "I have previously provided this example in R and posted it on GitHub:\n",
    "\n",
    "* [Linear Regression in R](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.R)\n",
    "* [Linear Regression in R markdown](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.Rmd) with docs \n",
    "* [Linear Regression in R document](https://github.com/GeostatsGuy/geostatsr/blob/master/linear_regression_demo_v2.html) knit as an HTML document\n",
    "\n",
    "and also in Excel:\n",
    "\n",
    "* [Linear Regression in Excel](https://github.com/GeostatsGuy/ExcelNumericalDemos/blob/master/Linear_Regression_Demo_v2.xlsx)\n",
    "\n",
    "#### Workflow Goals\n",
    "\n",
    "Learn the basics of time series analysis in Python to for analysis, modeling and prediction with production data. This includes:\n",
    "\n",
    "* Basic Python workflows and data preparation\n",
    "\n",
    "* Training linear regression model parameters to training data\n",
    "\n",
    "* Model visualizaton and checking  \n",
    "\n",
    "#### Getting Started\n",
    "\n",
    "Here's the steps to get setup in Python with the GeostatsPy package:\n",
    "\n",
    "1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/). \n",
    "2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal. \n",
    "3. In the terminal type: pip install geostatspy. \n",
    "4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality. \n",
    "\n",
    "You may want to copy the data file to your working directory. They are available here:\n",
    "\n",
    "* Tabular data - [Density_Por_data.csv](https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/Density_Por_data.csv).\n",
    "\n",
    "or you can use the code below to load the data directly from my GitHub [GeoDataSets](https://github.com/GeostatsGuy/GeoDataSets) repository.\n",
    "\n",
    "#### Import Required Packages\n",
    "\n",
    "Let's import the GeostatsPy package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "supress_warnings = False\n",
    "import os                                                   # to set current working directory \n",
    "import numpy as np                                          # arrays and matrix math\n",
    "import scipy.stats as st                                    # statistical methods\n",
    "import pandas as pd                                         # DataFrames\n",
    "import matplotlib.colors as colors                          # color bar normalization                                          \n",
    "import matplotlib.pyplot as plt                             # for plotting\n",
    "from matplotlib.ticker import (MultipleLocator, AutoMinorLocator) # control of axes ticks\n",
    "cmap = plt.cm.inferno                                       # default color bar, no bias and friendly for color vision defeciency\n",
    "plt.rc('axes', axisbelow=False)                              # grid behind plotting elements\n",
    "seed = 73073                                                # random number seed\n",
    "if supress_warnings == True:\n",
    "    import warnings                                         # supress any warnings for this demonstration\n",
    "    warnings.filterwarnings('ignore')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs. \n",
    "\n",
    "#### Declare functions\n",
    "\n",
    "Let's define a couple of functions to streamline plotting correlation matrices and visualization of a decision tree regression model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_grid():\n",
    "    plt.gca().grid(True, which='major',linewidth = 1.0); plt.gca().grid(True, which='minor',linewidth = 0.2) # add y grids\n",
    "    plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)\n",
    "    plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the Working Directory\n",
    "\n",
    "I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#os.chdir(\"C:\\PGE337\")                                       # set the working directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Data\n",
    "\n",
    "Let's load the provided dataset. 'Density_Por_data.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 density measures ($\\frac{g}{cm^3}$) and porosity measures from 2 rock units from the subsurface, porosity (as a fraction). We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it by printing a slice and by utilizing the 'head' DataFrame member function (with a nice and clean format, see below).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "add_error = False                                         # add random error to the response feature\n",
    "std_error = 1.0; seed = 71071\n",
    "\n",
    "yname = 'Porosity'; xname = 'Density'                     # specify the predictor features (x2) and response feature (x1)\n",
    "xmin = 1.0; xmax = 2.5                                    # set minumums and maximums for visualization \n",
    "ymin = 0.0; ymax = 25.0\n",
    "xlabel = 'Porosity'; ylabel = 'Density'                   # specify the feature labels for plotting\n",
    "yunit = '%'; xunit = '$g/cm^{3}$'\n",
    "Xlabelunit = xlabel + ' (' + xunit + ')'\n",
    "ylabelunit = ylabel + ' (' + yunit + ')'\n",
    "\n",
    "#df = pd.read_csv(\"Density_Por_data.csv\")                 # load the data from local current directory\n",
    "df = pd.read_csv(r\"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/Density_Por_data.csv\") # load the data from my github repo\n",
    "df = df.sample(frac=.30, random_state = 73073); df = df.reset_index() # extract 30% random to reduce the number of data\n",
    "\n",
    "if add_error == True:                                     # method to add error\n",
    "    np.random.seed(seed=seed)                             # set random number seed\n",
    "    df[yname] = df[yname] + np.random.normal(loc = 0.0,scale=std_error,size=len(df)) # add noise\n",
    "    values = df._get_numeric_data(); values[values < 0] = 0 # set negative to 0 in a shallow copy ndarray\n",
    "    \n",
    "dfy = pd.DataFrame(df[yname])                             # extract selected features as X and y DataFrames\n",
    "dfx = pd.DataFrame(df[xname])\n",
    "df = pd.concat([dfx,dfy],axis=1)                          # make one DataFrame with both X and y (remove all other features)\n",
    "\n",
    "y = df[yname].values.reshape(len(df))\n",
    "x = df[xname].values.reshape(len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Density</th>\n",
       "      <td>32.0</td>\n",
       "      <td>1.719994</td>\n",
       "      <td>0.262314</td>\n",
       "      <td>0.996736</td>\n",
       "      <td>1.547192</td>\n",
       "      <td>1.770422</td>\n",
       "      <td>1.838704</td>\n",
       "      <td>2.331653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Porosity</th>\n",
       "      <td>32.0</td>\n",
       "      <td>12.317525</td>\n",
       "      <td>3.224611</td>\n",
       "      <td>4.968240</td>\n",
       "      <td>10.492582</td>\n",
       "      <td>11.341744</td>\n",
       "      <td>14.459041</td>\n",
       "      <td>20.964941</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          count       mean       std       min        25%        50%  \\\n",
       "Density    32.0   1.719994  0.262314  0.996736   1.547192   1.770422   \n",
       "Porosity   32.0  12.317525  3.224611  4.968240  10.492582  11.341744   \n",
       "\n",
       "                75%        max  \n",
       "Density    1.838704   2.331653  \n",
       "Porosity  14.459041  20.964941  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().transpose()                                   # summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear Regression Model\n",
    "\n",
    "Let's first calculate the linear regression model with SciPy package, stats module.\n",
    "\n",
    "* recall we imported the module as 'st' above\n",
    "\n",
    "````python\n",
    "import scipy.stats as st                                    # statistical methods\n",
    "````\n",
    "\n",
    "We instantiate and fit the model all in one line of code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model parameters are, slope (b1) = -10.3, and the intercept (b0) = 30.03\n"
     ]
    }
   ],
   "source": [
    "slope, intercept, r_value, p_value, std_err = st.linregress(x,y) # instantiate and fit a linear regression model\n",
    "\n",
    "print('The model parameters are, slope (b1) = ' + str(round(slope,2)) + ', and the intercept (b0) = ' + str(round(intercept,2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have 5 outputs when we instantiate and fit our model.\n",
    "\n",
    "* **slope** - the slope of our linear model, the $b_1$ in the model, $y = b_1 x + b_0$\n",
    "* **intercept** - the intercept of our linear model, the $b_0$ in the model, $y = b_1 x + b_0$\n",
    "* **r_value** - the Pearson correlation, the square is the $r^2$, the variance explained\n",
    "* **p_value** - the p-value for the hypothesis test for the slope of the model of zero\n",
    "* **stderr** - the standard error of the slope parameter, $SE_{b_1}$\n",
    "\n",
    "Let's plot the data and the model, to get out estimates we substitute our predictor feature values into our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 60752 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_values = np.linspace(xmin,xmax,100)                       # return an array of density values \n",
    "y_model = slope * x_values + intercept                      # apply our linear regression model to estimate at the training data values\n",
    "\n",
    "plt.subplot(111)                                            # plot the model\n",
    "plt.plot(x, y, 'o', label='data', color = 'darkorange', alpha = 0.2, markeredgecolor = 'black',zorder=10)\n",
    "plt.plot(x_values, y_model, label='model', color = 'black',zorder=1)\n",
    "plt.title('Linear Regression Model, Regression of ' + yname + ' on ' + xname)\n",
    "plt.xlabel(xname + ' (' + xunit + ')')\n",
    "plt.ylabel(yname + ' (' + yunit + ')')\n",
    "plt.legend(); add_grid(); plt.xlim([xmin,xmax]); plt.ylim([ymin,ymax])\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.0, wspace=0.2, hspace=0.2); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the Solution Space\n",
    "\n",
    "Sample of mesh of possible linear regression model parameters and calculate the MSE over all data.\n",
    "\n",
    "* Note we are not attemption train and test split for model hyperparameter tuning. While we could add a hyper parameter through regularization with LASSO or ridge regression, we just want to visualize this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 88812 characters]",
      "text/plain": [
       "<Figure size 640x480 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vmin = 1.5; vmax = 20.0; #vmax = np.max(MSE_mat)            # set min and max MSE for visualization\n",
    "norm = colors.Normalize(vmin=vmin, vmax=vmax)\n",
    "nstep = 100; MSE_mat = np.zeros([nstep,nstep])              # set up the madel parameter mesh\n",
    "sb0 = -60.0; eb0 = 100.0; stepb0 = (eb0-sb0)/nstep;         \n",
    "sb1 = -80.0; eb1 = 60.0; stepb1 = (eb1-sb1)/nstep;\n",
    "b0_vector = np.arange(sb0, eb0, stepb0); b1_vector = np.arange(eb1, sb1, -1*stepb1)\n",
    "b0_mat, b1_mat = np.meshgrid(b0_vector, b1_vector)\n",
    "\n",
    "for ib0, b0 in enumerate(b0_vector):                        # calculate the MSE for all possible model parameters\n",
    "    for ib1, b1 in enumerate(b1_vector):\n",
    "        y_hat = b1 * x + b0\n",
    "        MSE_mat[ib1,ib0] = np.sqrt(((y - y_hat) ** 2).mean())\n",
    "\n",
    "plt.subplot(131)\n",
    "im = plt.imshow(b0_mat,cmap=plt.cm.inferno)\n",
    "plt.title('$b_0$ Mesh')\n",
    "plt.xlabel('$b_0$'); plt.ylabel('$b_1$')\n",
    "cbar = plt.colorbar(im, orientation=\"vertical\", ticks=np.linspace(sb0, eb0, 10))\n",
    "cbar.set_label('$b_0$', rotation=270, labelpad=20)\n",
    "\n",
    "plt.subplot(132)\n",
    "im = plt.imshow(b1_mat,cmap=plt.cm.inferno)\n",
    "plt.title('$b_1$ Mesh')\n",
    "plt.xlabel('$b_0$'); plt.ylabel('$b_1$')\n",
    "cbar = plt.colorbar(im, orientation=\"vertical\", ticks=np.linspace(sb1, eb1, 10))\n",
    "cbar.set_label('$b_1$', rotation=270, labelpad=20)\n",
    "\n",
    "plt.subplot(133)\n",
    "im = plt.imshow(MSE_mat,interpolation = None,extent = [sb0,eb0,sb1,eb1], vmin = vmin, vmax = vmax,alpha=1.0,cmap = plt.cm.inferno_r)\n",
    "plt.scatter(intercept,slope,marker='x',color='black')\n",
    "plt.title('MSE Mesh and Optimal Model Parameters')\n",
    "plt.xlabel('b0'); plt.ylabel('b1')\n",
    "cbar = plt.colorbar(im, orientation=\"vertical\", ticks=np.linspace(vmin, vmax, 10))\n",
    "cbar.set_label('MSE', rotation=270, labelpad=20)\n",
    "\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.0, wspace=0.2, hspace=0.1); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualize the Solution Space and Sample Suboptimal solutions.\n",
    "\n",
    "We produce the follow 2 plots:\n",
    "    \n",
    "1. the MSE mapped and contoured over a wide range of model parameters, $b_1$ slope and $b_0$ intercept. Plot the optimal (+) and suboptimal model parameter combinations (o) (right plot)\n",
    "\n",
    "2. the data and the optimal and sampled suboptimal models colored by MSE (same as right plot) and sorted with descending MSE so the best models are on top\n",
    "\n",
    "Suboptimal models are sampled with inverse weighting such that better model parameters (minimize MSE) are more likely sampled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1469916 characters]",
      "text/plain": [
       "<Figure size 640x480 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nsamples = 200\n",
    "\n",
    "np.random.seed(seed=seed)\n",
    "df_results = pd.DataFrame({'b0':b0_mat.flatten(),'b1':b1_mat.flatten(),'MSE':MSE_mat.flatten()})\n",
    "\n",
    "samples = df_results.sample(weights=(1.0/df_results['MSE'])**2.0, n=nsamples)\n",
    "samples = samples.sort_values(by='MSE',ascending=False).reset_index()\n",
    "\n",
    "fig, ax = plt.subplots(1,2, gridspec_kw={'width_ratios': [2, 1]})\n",
    "\n",
    "ax[0].plot(x, y, 'o', label='data', color = 'white', alpha = 1.0, markeredgecolor = 'black',zorder=10000)\n",
    "ax[0].plot(x_values, y_model, label='model', color = 'black',ls='--',lw=3,zorder=10000)\n",
    "\n",
    "for isample in range(0,len(samples)):\n",
    "    y_alt = samples.loc[isample,'b1'] * x_values + samples.loc[isample,'b0']\n",
    "    ax[0].plot(x_values, y_alt, c = 'black',zorder=isample*3+1,lw=3)\n",
    "    ip = ax[0].plot(x_values, y_alt, c = plt.cm.inferno_r(norm(samples.loc[isample,'MSE'])),zorder=isample*10+2,lw=2)\n",
    "\n",
    "ax[0].set_title('Data, Linear Regression Model, and Suboptimal Models')\n",
    "ax[0].set_xlabel(xname + ' (' + xunit + ')')\n",
    "ax[0].set_ylabel(yname + ' (' + yunit + ')')\n",
    "ax[0].set_xlim([xmin,xmax]); ax[0].set_ylim([ymin,ymax])\n",
    "\n",
    "im = ax[1].imshow(MSE_mat,interpolation = None,extent = [sb0,eb0,sb1,eb1], vmin = vmin, vmax = vmax,alpha=1.0,cmap = plt.cm.inferno_r)\n",
    "ax[1].contour(b0_mat,b1_mat,MSE_mat,levels = np.logspace(0.0,1.1,10),colors='black',alpha=0.7)\n",
    "ax[1].set_title('Linear Regression MSE and Suboptimal Models')\n",
    "ax[1].set_xlabel('b0')\n",
    "ax[1].set_ylabel('b1')\n",
    "cbar = plt.colorbar(im, orientation=\"vertical\", ticks=np.linspace(vmin, vmax, 10))\n",
    "cbar.set_label('MSE', rotation=270, labelpad=20)\n",
    "# plt.savefig(fig_name + \".\" + image_type, dpi=dpi)\n",
    "plt.scatter(intercept,slope,marker='x',color='black')\n",
    "for isample in range(0,len(samples)):\n",
    "    ax[1].scatter(samples.loc[isample,'b0'],samples.loc[isample,'b1'],marker='o',edgecolor='white',color = plt.cm.inferno_r(norm(samples.loc[isample,'MSE'])),zorder=100)\n",
    "plt.grid()\n",
    "\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.0, wspace=0.1, hspace=0.1); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very interesting summary of the optimal and many solutions with variable level of suboptimality.\n",
    "\n",
    "#### Comments\n",
    "\n",
    "Linear regression is efficient with the stats module of the SciPy package in Python. With one line of code we can build a model and get the outputs needed to make predictions and check the model. I hope this was helpful,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "On twitter I'm the @GeostatsGuy.\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "#### More on Michael Pyrcz and the Texas Center for Geostatistics:\n",
    "\n",
    "### Michael Pyrcz, Professor, University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
