{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3672fa62",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Interactive Bayesian Linear Regression with Metropolis-Hastings  \n",
    "\n",
    "### Michael J. Pyrcz, Professor, The University of Texas at Austin \n",
    "\n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c708d65d",
   "metadata": {},
   "source": [
    "This is an interactive demonstration of Bayesian linear regression. First, here's some details:\n",
    "\n",
    "#### Bayesian Updating\n",
    "\n",
    "The frequentist formulation of the linear regression model is: \n",
    "\n",
    "\\begin{equation}\n",
    "y = b_1 \\times x + b_0 + \\sigma\n",
    "\\end{equation}\n",
    "\n",
    "where $x$ is the predictor feature, $b_1$ is the slope parameter, $b_0$ is the intercept parameter and $\\sigma$ is the error or noise. There is an analytical form for the ordinary least squares solution to fit the available data while minimizing the L2 norm of the data error vector.\n",
    "\n",
    "For the Bayesian formulation of linear regression is we pose the model as a prediction of the distribution of the response, $Y$, now a random variable:\n",
    "\n",
    "\\begin{equation}\n",
    "Y \\sim N(\\beta^{T}X, \\sigma^{2} I)\n",
    "\\end{equation}\n",
    "\n",
    "We estimate the model parameter distributions through Bayesian updating for infering the model parameters from a prior and likelihood from training data.\n",
    "\n",
    "\\begin{equation}\n",
    "p(\\beta | y, X) = \\frac{p(y,X| \\beta) p(\\beta)}{p(y,X)}\n",
    "\\end{equation}\n",
    "\n",
    "In general for continuous features we are not able to directly calculate the posterior and we must use a sampling method, such as Markov chain Monte Carlo (McMC) to sample the posterior.\n",
    "\n",
    "For a complete lecture with linked Python workflows check out:\n",
    "\n",
    "* [Bayesian Linear Regression Lecture](https://www.youtube.com/watch?v=LzZ5b3wdZQk&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=33)\n",
    "\n",
    "* [Bayesian Probability Lecture](https://www.youtube.com/watch?v=Ppwfr8H177M&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=6)\n",
    "\n",
    "Here's my McMC Metropolis-Hastings workflow with more details:\n",
    "\n",
    "* [Bayesian Linear Regression Workflow](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceMachineLearning_Simple_Bayesian_Linear_Regression.ipynb)\n",
    "\n",
    "and here's my Bayesian linear regression workflow with the pymc3 Python package:\n",
    "\n",
    "* [Bayesian Linear Regression Workflow with the pymc3 Python Package](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/SubsurfaceDataAnalytics_BayesianRegression.ipynb)\n",
    "\n",
    "#### Bayesian Linear Regression with the Metropolis-Hastings Sampler\n",
    "\n",
    "Here's the basic steps of the Metropolis-Hastings Sampler: \n",
    "\n",
    "For $\\ell = 1, \\ldots, L$:\n",
    "\n",
    "1. Assign random values for the the initial sample of model parameters, $\\beta(\\ell = 1) = b_1(\\ell = 1)$, $b_0(\\ell = 1)$ and $\\sigma^2(\\ell = 1)$. \n",
    "2. Propose new model parameters based on a proposal function, $\\beta^{\\prime} = b_1$, $b_0$ and $\\sigma^2$. \n",
    "3. Calculate probability of acceptance of the new proposal, as the ratio of the posterior probability of the new model parameters given the data to the previous model parameters given the data multiplied by the probability of the old step given the new step divided by the probability of the new step given the old. \n",
    "\n",
    "\\begin{equation}\n",
    "P(\\beta \\rightarrow \\beta^{\\prime}) = min\\left(\\frac{p(\\beta^{\\prime}|y,X) }{ p(\\beta | y,X)} \\cdot \\frac{p(\\beta^{\\prime}|\\beta) }{ p(\\beta | \\beta^{\\prime})},1\\right)\n",
    "\\end{equation}\n",
    "\n",
    "4. Apply Monte Carlo simulation to conditionally accept the proposal, if accepted, $\\ell = \\ell + 1$, and sample $\\beta(\\ell) = \\beta^{\\prime}$\n",
    "5. Go to step 2.\n",
    "\n",
    "Let's talk about this system. First the left hand side:\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{p(\\beta^{\\prime}|y,X) }{ p(\\beta | y,X)}\n",
    "\\end{equation}\n",
    "\n",
    "We are calculating the ratio of the posterior probability (likelihood times prior) of the model parameters given the data and prior model for proposed sample over the current sample. \n",
    "\n",
    "* As you will see below it is quite practical to calculate this ratio.\n",
    "* If the proposed sampled is more likely than the current sample, we will have a value greater than 1.0, it will truncate to 1.0 and we accept the proposed sample.\n",
    "* If the proposed sample is less likely than the current sample, we will have a value less than 1.0, then we will use Monte Carlo sampling to randomly choice the proposed sample with this probability of acceptance.\n",
    "\n",
    "This proceedure allows us to walk through the model parameter space and sample the parameters with the current rates, after the burn-in chain.\n",
    "\n",
    "Now, what about this part of the equation?\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{p(\\beta^{\\prime}|\\beta) }{ p(\\beta | \\beta^{\\prime})}\n",
    "\\end{equation}\n",
    "\n",
    "There is a problem with this procedure if we use asymmetric probability distributions for the model parameters! \n",
    "\n",
    "* E.g. for example, if we use a positively skewed distribution (e.g., log normal) then we are more likely to step to larger values due to this distribution, and not due to the prior nor the likelihood.\n",
    "\n",
    "* This term removes this bias, so that we have fair samples!\n",
    "\n",
    "You will see below that we remove this issue by assuming symmetric model parameter distributions, even though many use  na assymetric gamma distribution for sigma, given it cannot have negative values.\n",
    "\n",
    "* My goal is the simplest possible demonstration.\n",
    "\n",
    "#### Our Simplified Demonstration Metropolis Sampling\n",
    "\n",
    "Let's further specify this workflow for our simple demonstration. \n",
    "\n",
    "1. I have assumed a Gaussian, symmetric distribution for all model parameters as a result this relationship holds for all possible model parameter, current and proposed.\n",
    "\n",
    "\\begin{equation}\n",
    "\\frac{p(\\beta^{\\prime}|\\beta) }{ p(\\beta | \\beta^{\\prime})} = 1.0\n",
    "\\end{equation}\n",
    "\n",
    "    So we now have this simplified probability of proposal acceptance, note this is know as Metropolis Sampling.\n",
    "    \n",
    "\\begin{equation}\n",
    "P(\\beta \\rightarrow \\beta^{\\prime}) = min \\left( \\frac{p(\\beta^{\\prime}|y,X) }{ p(\\beta | y,X)},1 \\right) \n",
    "\\end{equation}\n",
    "\n",
    "2. Now, let's substitute our Bayesian formulation for our Bayesian linear regression model.\n",
    "\n",
    "\\begin{equation}\n",
    "p(\\beta^{\\prime} | y, X) = \\frac{p(y,X| \\beta^{\\prime}) p(\\beta^{\\prime})}{p(y,X)} \\quad \\text{      and        } \\quad p(\\beta | y, X) = \\frac{p(y,X| \\beta) p(\\beta)}{p(y,X)}\n",
    "\\end{equation}\n",
    "\n",
    "    If we substitute these into our probability of acceptance above we get this.\n",
    "\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\beta \\rightarrow \\beta^{\\prime}) = min \\left( \\frac{p(\\beta^{\\prime}|y,X) }{ p(\\beta | y,X)},1 \\right) = min \\left( \\frac{ \\left( \\frac{p(y,X| \\beta_{new}) p(\\beta_{new}) } {p(y,X)} \\right) }{ \\left( \\frac{ p(y,X| \\beta) p(\\beta)}{p(y,X)} \\right) },1 \\right)\n",
    "\\end{equation}\n",
    "\n",
    "    Note that the evidence terms cancel out.\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\beta \\rightarrow \\beta^{\\prime}) = min \\left( \\frac{ p(y,X| \\beta_{new}) p(\\beta_{new}) }{ p(y,X| \\beta) p(\\beta)},1 \\right)\n",
    "\\end{equation}\n",
    "\n",
    "    Since we are working with a likelihood ratio, we can work with densities instead of probabilities.\n",
    "\n",
    "\\begin{equation}\n",
    "P(\\beta \\rightarrow \\beta^{\\prime}) = min \\left( \\frac{ f(y,X| \\beta_{new}) f(\\beta_{new}) }{ f(y,X| \\beta) f(\\beta) } ,1 \\right)\n",
    "\\end{equation}\n",
    "\n",
    "3. Finally for improved solution stability we can calculate the natural log ratio:\n",
    "\n",
    "\\begin{equation}\n",
    "ln(P(\\beta \\rightarrow \\beta^{\\prime})) = min \\left( ln \\left[ \\frac{ f(y,X| \\beta_{new}) f(\\beta_{new}) }{ f(y,X| \\beta) f(\\beta) } \\right],0 \\right) = min \\left( \\left[ln(f(y,X| \\beta_{new})) + ln(f(\\beta_{new})) \\right] - \\left[ ln(f(y,X| \\beta)) + ln(f(\\beta)) \\right],0 \\right)\n",
    "\\end{equation}\n",
    "\n",
    "4. We calculate probability of proposal acceptance, as exponentiation of the above. \n",
    "\n",
    "5. How do we calculate the likelihood density? If we assume independence between all of the data we can take the product sum of the probabilities (densities) of all the response values given the predictor and model parameter sample! Given, the Gaussian assumption for the response feature, we can calculate the densities for each data from the Gaussian PDF.\n",
    "\n",
    "\\begin{equation}\n",
    "f_{y,X | \\beta}(y) \\sim N [ b_1 \\cdot X + b_0, \\sigma ]\n",
    "\\end{equation}\n",
    "\n",
    "    and under the assumption of indepedence we can take the produce sum over all training data.\n",
    "\n",
    "\\begin{equation}\n",
    "f(y,X| \\beta) = \\prod_{\\alpha = 1}^{n} f_{y,X | \\beta}(y_{\\alpha})\n",
    "\\end{equation}\n",
    "\n",
    "Note, this workflow was developed with assistance from Fortunato Nucera's Medium Article [Mastering Bayesian Linear Regression from Scratch: A Metropolis-Hastings Implementation in Python](https://medium.com/@tinonucera/bayesian-linear-regression-from-scratch-a-metropolis-hastings-implementation-63526857f191). I highly recommend this accessible description and demonstration. Thank you, Fortunato.\n",
    "\n",
    "#### Load and Configure the Required Libraries\n",
    "\n",
    "The following code loads the required libraries and sets a plotting default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "09e9ba4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "supress_warnings = True                                     # supress warnings?\n",
    "import numpy as np                                          # arrays and matrix math\n",
    "import pandas as pd                                         # DataFrames\n",
    "import matplotlib.pyplot as plt                             # plotting\n",
    "from matplotlib.patches import Rectangle                    # build a custom legend\n",
    "from matplotlib.ticker import (MultipleLocator, AutoMinorLocator) # control of axes ticks\n",
    "from matplotlib.gridspec import GridSpec                    # nonstandard subplots\n",
    "from matplotlib.patches import Ellipse                      # plot prior model\n",
    "import scipy.stats as stats                                 # parametric distributions\n",
    "from sklearn.linear_model import LinearRegression           # frequentist model for comparison\n",
    "from ipywidgets import interactive                          # widgets and interactivity\n",
    "from ipywidgets import widgets                            \n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import Label\n",
    "from ipywidgets import VBox, HBox\n",
    "cmap = plt.cm.inferno                                       # default color bar, no bias and friendly for color vision defeciency\n",
    "plt.rc('axes', axisbelow=True)                              # grid behind plotting elements\n",
    "if supress_warnings == True:\n",
    "    import warnings                                         # supress any warnings for this demonstration\n",
    "    warnings.filterwarnings('ignore')  \n",
    "plt.rc('axes', axisbelow=True)                              # set axes and grids in the background for all plots\n",
    "seed = 73073                                                # random number seed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f9b1c7",
   "metadata": {},
   "source": [
    "#### Declare Functions\n",
    "\n",
    "The following functions include:\n",
    "    \n",
    "* **next_proposal** - propose a next model parameters from previous previous model parameters, this is the proposal method, parameterized by step standarrd deviation and Gaussian distribution.\n",
    "\n",
    "* **likelihood_density** - calculate the product of all the densities for all the data given the model parameters. Since we are working with the log densities, we sum over all the data.\n",
    "\n",
    "* **prior_density** - calculate the product of the densities for all the model parameters given the prior model. Since we are working with the log densities, we sum over all the model parameters. This is an assumption of independence.\n",
    "\n",
    "* **add_grid** - improved grid for the plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "678335dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def next_proposal(prev_theta, step_stdev = 0.5):            # assuming a Gaussian distribution centered on previous theta and step stdev  \n",
    "    out_theta = stats.multivariate_normal(mean=prev_theta,cov=np.eye(3)*step_stdev**2).rvs(1)\n",
    "    return out_theta\n",
    "\n",
    "def likelihood_density(x,y,theta):                          # likelihood - probability (density) of the data given the model\n",
    "    density = np.sum(stats.norm.logpdf(y, loc=theta[0]*x+theta[1],scale=theta[2])) # assume independence, sum is product in log space\n",
    "    return density\n",
    "\n",
    "def prior_density(theta,prior):                             # prior - probability (density) of the model parameters given the prior \n",
    "    mean = np.array([prior[0,0],prior[1,0],prior[2,0]]); cov = np.zeros([3,3]); cov[0,0] = prior[0,1]; cov[1,1] = prior[1,1]; cov[2,2] = prior[2,1]\n",
    "    prior_out = stats.multivariate_normal.logpdf(theta,mean=mean,cov=cov,allow_singular=True)\n",
    "    return prior_out\n",
    "\n",
    "def add_grid():\n",
    "    plt.gca().grid(True, which='major',linewidth = 1.0); plt.gca().grid(True, which='minor',linewidth = 0.2) # add y grids\n",
    "    plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)\n",
    "    plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2aae770",
   "metadata": {},
   "source": [
    "#### Build a Dataset with Known Truth Model Parameters for Linear Regression\n",
    "\n",
    "Let's build a simple dataset with known linear regresin parameters, $b_1$ is the slope parameter, $b_0$ is the intercept parameter and $\\sigma$ is the error or noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "55a1c55a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 60724 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(seed = seed)                                 # set random number seed\n",
    "\n",
    "data_b1 = 3; data_b0 = 20; data_sigma = 5; n = 100          # set data model parameters\n",
    "\n",
    "x = np.random.rand(n)*30                                    # random x values\n",
    "y = data_b1*x+data_b0 + np.random.normal(loc=0.0,scale=data_sigma,size=n) # y as a linear function of x + random noise \n",
    "\n",
    "xhat = np.linspace(-10,40,100)                              # set of x values to predict and visualize the model\n",
    "linear_model = LinearRegression().fit(x.reshape(-1, 1),y)   # instantiate and train the frequentist linear regression model\n",
    "yhat = linear_model.predict(xhat.reshape(-1, 1))            # make predictions for model plotting\n",
    "\n",
    "plt.subplot(111)\n",
    "plt.scatter(x, y,c='red',s=10,edgecolor='black')\n",
    "plt.plot(xhat,yhat,c='red'); add_grid()\n",
    "plt.xlabel(\"Predictor Feature\"); plt.ylabel(\"Response Feature\"); plt.title('Data and Linear Regression Model')\n",
    "plt.gca().add_patch(Rectangle((1.5,93.0),4.3,23,facecolor='white',edgecolor='black',linewidth=0.5))\n",
    "plt.annotate('$b_1$ = ' + str(np.round(linear_model.coef_[0],2)),[2,110])\n",
    "plt.annotate('$b_0$ = ' + str(np.round(linear_model.intercept_,2)),[2,103])\n",
    "plt.annotate('$\\sigma$ = ' + str(np.round(data_sigma,2)),[2,96])\n",
    "plt.xlim([0,30]); plt.ylim([0,120])\n",
    "\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.5); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5222fe81",
   "metadata": {},
   "source": [
    "#### Assume the Prior Model\n",
    "\n",
    "We assume a multivariate Gaussian distribution for the slope and intercept parameters, $f_{b_1,b_0,\\sigma}(b_1,b_0,\\sigma)$ with indepedence between $b_1$, $b_0$, and $\\sigma$.\n",
    "\n",
    "* For a naive prior assume a very large standard deviation.\n",
    "We will work in log probability and log density to improve stability of the system. \n",
    "\n",
    "* We want to avoid product sums of a many values near zero as the the probabilities will disappear due to computer floating point precision. \n",
    "* In log space, we can add calculate probabilities of independent events by summing (instead of multiplication) these negative values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d3750f8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "prior = np.zeros([3,2])                                     # prior distributions\n",
    "prior[0,:] = [4.0,1.0]                                      # Gaussian prior model for slope, mean and standard deviation\n",
    "prior[1,:] = [13.0,3.0]                                     # Gaussian prior model for intercept, mean and standard deviation\n",
    "prior[2,:] = [7.0,1.0]                                      # Gaussian prior model for sigma, k (shape) and phi (scale), recall mean = k x phi, var = k x phi^2 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061571e6",
   "metadata": {},
   "source": [
    "#### Bayesian Linear Regression with McMC Metropolis-Hastings \n",
    "\n",
    "The Bayesian linear regression with McMC Metropolis-Hastings workflow.\n",
    "\n",
    "1. assign an random initial set of model parameters\n",
    "2. apply a proposal rule to assign a new set of model parameters given the previous set of model paramters\n",
    "3. calculate the ratio of the likelihood of the new model parameters over the previous model parameters given the data\n",
    "4. conditionally accept the proposal based on this ratio, i.e., if proposal is more like accept it, if less likely with a probability based on the ratio.\n",
    "5. goto to 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cb53b91d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "l = widgets.Text(value='               Interactive Bayesian Linear Regression with McMC Metropolis-Hastings Demo, Prof. Michael Pyrcz, The University of Texas at Austin',\n",
    "                 layout=Layout(width='890px', height='30px'))\n",
    "\n",
    "max_accepted = widgets.IntSlider(min=2, max = 5001, value=100, step = 20, description = '$\\ell$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "# radius = widgets.FloatSlider(min=10, max = 500, value=110, step = 10, description = r'$r$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "# minpts = widgets.IntSlider(min=2, max = 20, value=4, step = 1, description = r'$min_{pts}$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)\n",
    "\n",
    "ui = widgets.HBox([max_accepted],)\n",
    "ui2 = widgets.VBox([l,ui],)\n",
    "\n",
    "def run_plot(max_accepted):\n",
    "\n",
    "    np.random.seed(seed = seed)\n",
    "    step_stdev = 0.2\n",
    "    \n",
    "    thetas = np.random.rand(3).reshape(1,-1) # seed a random first step\n",
    "    accepted = 0\n",
    "    \n",
    "    while accepted < max_accepted:\n",
    "        theta_new = next_proposal(thetas[-1,:],step_stdev=step_stdev) # next proposal\n",
    "        \n",
    "        log_like_new = likelihood_density(x,y,theta_new)        # new and prior likelihoods, log of density\n",
    "        log_like = likelihood_density(x,y,thetas[-1,:])\n",
    "            \n",
    "        log_prior_new = prior_density(theta_new,prior)          # new and prior prior, log of density\n",
    "        log_prior = prior_density(thetas[-1,:],prior)\n",
    "        \n",
    "        likelihood_prior_proposal_ratio = np.exp((log_like_new + log_prior_new) - (log_like + log_prior)) # calculate log ratio\n",
    "    \n",
    "        if likelihood_prior_proposal_ratio > np.random.rand(1): # conditionally accept by likelihood ratio\n",
    "            thetas = np.vstack((thetas,theta_new)); accepted += 1\n",
    "    \n",
    "    df = pd.DataFrame(np.vstack([thetas[:,0],thetas[:,1],thetas[:,2]]).T, columns= ['Slope','Intercept','Sigma'])\n",
    "    \n",
    "    fig = plt.figure(constrained_layout=False)\n",
    "    gs = GridSpec(2, 2, figure=fig)\n",
    "        \n",
    "    ax1 = fig.add_subplot(gs[:, 0])\n",
    "    \n",
    "    burn_chain = 250\n",
    "    alpha = 0.1\n",
    "    max_sample = 1000\n",
    "    viz_buff = 50\n",
    "    \n",
    "    alpha_burn = np.arange(0,burn_chain,dtype='float')\n",
    "    alpha_burn = 1+(alpha_burn - max_accepted)/viz_buff\n",
    "    alpha_burn = np.where(alpha_burn<0, 0, alpha_burn)\n",
    "    alpha_burn = alpha_burn[alpha_burn <= 1.0]\n",
    "\n",
    "    ax1.scatter(prior[0,0],prior[1,0],color='black',marker='x',s=30)\n",
    "    ax1.annotate('Prior',[prior[0,0]+0.1,prior[1,0]+0.1],color='black')    \n",
    "    ell = Ellipse(xy=(prior[0,0],prior[1,0]),width=prior[0,1], height=prior[1,1],angle=0.0,ls='--') \n",
    "    ell.set_edgecolor('black'); ell.set_facecolor('none')\n",
    "    ax1.add_artist(ell)\n",
    "    ax1.scatter(linear_model.coef_[0],linear_model.intercept_,color='black',marker='x',s=30)\n",
    "    ax1.annotate('OLS',[linear_model.coef_[0]+0.1,linear_model.intercept_+0.1],color='black')\n",
    "    ax1.scatter(thetas[:burn_chain,0],thetas[:burn_chain,1],s=20,marker = 'o',c='black',edgecolor='black',alpha=alpha_burn,linewidth=1.0,cmap=plt.cm.inferno,zorder=10)\n",
    "    if max_accepted > burn_chain:\n",
    "        ax1.scatter(thetas[burn_chain:,0],thetas[burn_chain:,1],s=30,c=np.arange(burn_chain,max_accepted+1,1),alpha=1.0,edgecolor='black',linewidth=0.1,cmap=plt.cm.inferno,zorder=10)\n",
    "    ax1.scatter(thetas[-1,0],thetas[-1,1],s=50,c='white',alpha=1.0,edgecolor='black',linewidth=1.0,cmap=plt.cm.inferno,zorder=100)\n",
    "    ax1.plot(thetas[:burn_chain,0],thetas[:burn_chain,1],color='black',linewidth=1.0,zorder=1)\n",
    "    add_grid(); ax1.set_xlabel('Slope, $b_1$'); ax1.set_ylabel('Intercept, $b_0$'); ax1.set_title('McMC Samples Bayesian Linear Regression') \n",
    "    ax1.set_xlim([0,5]); ax1.set_ylim([0,25])\n",
    "    \n",
    "    ax2 = fig.add_subplot(gs[0, 1])\n",
    "    ax2.plot(np.arange(0,accepted+1,1),thetas[:,0],c='red',zorder=100) \n",
    "    ax2.scatter(accepted,thetas[-1,0],s=50,c='white',alpha=1.0,edgecolor='black',linewidth=1.0,cmap=plt.cm.inferno,zorder=100)\n",
    "    ax2.set_xlabel('McMC Metropolis-Hastings Sample'); ax2.set_ylabel(r'$b_1$'); ax2.set_title(\"McMC Slope Samples\"); add_grid()\n",
    "    ax2.plot([0,max_sample],[linear_model.coef_[0],linear_model.coef_[0]],c='darkred',lw=0.5,zorder=10)\n",
    "    ax2.annotate('Linear Regression',[max_sample*0.70,linear_model.coef_[0]-0.8],color='darkred')\n",
    "    ax2.plot([0,max_sample],[prior[0,0],prior[0,0]],c='black',zorder=10)\n",
    "    ax2.annotate('Prior Model',[max_sample*0.8,prior[0,0]-0.8])\n",
    "    lower = stats.norm.ppf(alpha/2.0,loc=prior[0,0],scale=prior[0,1])\n",
    "    ax2.plot([0,max_sample],[lower,lower],c='black',ls='--',lw=0.5,zorder=10)\n",
    "    upper = stats.norm.ppf(1-alpha/2.0,loc=prior[0,0],scale=prior[0,1])\n",
    "    ax2.plot([0,max_sample],[upper,upper],c='black',ls='--',lw=0.5,zorder=10)\n",
    "    ax2.fill_between([0,max_sample],[lower,lower],[upper,upper],color='black',alpha=0.05,zorder=1)\n",
    "    ax2.set_xlim([0,max_sample]); ax2.set_ylim([0,10])\n",
    "    \n",
    "    ax3 = fig.add_subplot(gs[1, 1])\n",
    "    ax3.plot(np.arange(0,accepted+1,1),thetas[:,1],c='red',zorder=100) \n",
    "    ax3.scatter(accepted,thetas[-1,1],s=50,c='white',alpha=1.0,edgecolor='black',linewidth=1.0,cmap=plt.cm.inferno,zorder=100)\n",
    "    ax3.set_xlabel('McMC Metropolis-Hastings Sample'); ax3.set_ylabel(r'$b_0$'); ax3.set_title(\"McMC Intercept Samples\"); add_grid()\n",
    "    ax3.plot([0,max_sample],[linear_model.intercept_,linear_model.intercept_],c='darkred',lw=0.5,zorder=10)\n",
    "    ax3.annotate('Linear Regression',[max_sample*0.70,linear_model.intercept_-2.2],color='darkred')\n",
    "    ax3.plot([0,max_sample],[prior[1,0],prior[1,0]],c='black',zorder=10)\n",
    "    ax3.annotate('Prior Model',[max_sample*0.8,prior[1,0]-2.2])\n",
    "    lower = stats.norm.ppf(alpha/2.0,loc=prior[1,0],scale=prior[1,1])\n",
    "    ax3.plot([0,max_sample],[lower,lower],c='black',ls='--',lw=0.5,zorder=10)\n",
    "    upper = stats.norm.ppf(1-alpha/2.0,loc=prior[1,0],scale=prior[1,1])\n",
    "    ax3.plot([0,max_sample],[upper,upper],c='black',ls='--',lw=0.5,zorder=10)\n",
    "    ax3.fill_between([0,max_sample],[lower,lower],[upper,upper],color='black',alpha=0.05,zorder=1)\n",
    "    ax3.set_xlim([0,max_sample]); ax3.set_ylim([0,30])\n",
    "    \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.0, wspace=0.2, hspace=0.5); plt.show()\n",
    "    \n",
    "# connect the function to make the samples and plot to the widgets    \n",
    "interactive_plot = widgets.interactive_output(run_plot, {'max_accepted':max_accepted})\n",
    "interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b79a1e8c",
   "metadata": {},
   "source": [
    "### Interactive Bayesian Linear Regression with McMC Metropolis-Hastings Demonstration \n",
    "\n",
    "#### Michael Pyrcz, Professor, The University of Texas at Austin "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "483a17df",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "08267c7f8d854a1197e0fad5c30bf501",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='               Interactive Bayesian Linear Regression with McMC Metropolis-Hastings…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3adb7afd5c34b679a8aa21fc94c4ae1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui2, interactive_plot)                           # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9db45c3",
   "metadata": {},
   "source": [
    "#### Comments\n",
    "\n",
    "This was an interactive demonstration of Bayesian Linear Regression with McMC Metropolis-Hastings Sampling.\n",
    "\n",
    "I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy. \n",
    "  \n",
    "I hope this was helpful,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "#### The Author:\n",
    "\n",
    "### Michael J Pyrcz, Professor, The University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
