{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Interactive Decision Tree from Predictive Machine Learning\n",
    "\n",
    "\n",
    "### Michael Pyrcz, Professor, University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "\n",
    "### The Interactive Workflow\n",
    "\n",
    "Here's a simple workflow, demonstration of decision trees for subsurface modeling workflows. This should help you get started with building subsurface models that data analytics and machine learning. \n",
    "\n",
    "* I have a lecture on [Decision Trees](https://youtu.be/JUGo1Pu3QT4) as part of my [Machine Learning](https://www.youtube.com/playlist?list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf) course. Note, all recorded lectures, interactive and well-documented workflow demononstrations are available on my GitHub repository [GeostatsGuy's Python Numerical Demos](https://github.com/GeostatsGuy/PythonNumericalDemos).\n",
    "\n",
    "* Here's some basic details about predictive machine learning decision tree models. \n",
    "\n",
    "### Decision Tree\n",
    "\n",
    "Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.\n",
    "\n",
    "**Prediction**\n",
    "\n",
    "* estimate a function $\\hat{f}$ such that we predict a response feature $Y$ from a set of predictor features $X_1,\\ldots,X_m$. \n",
    "\n",
    "* the prediction is of the form $\\hat{Y} = \\hat{f}(X_1,\\ldots,X_m)$ \n",
    "\n",
    "**Suppervised Learning**\n",
    "\n",
    "* the response feature label, $Y$, is available over the training and testing data\n",
    "    \n",
    "**Hiearchical, Binary Segmentation of the Feature Space**\n",
    "\n",
    "The fundamental idea is to divide the predictor space, $𝑋_1,\\ldots,X_m$, into $J$ mutually exclusive, exhaustive regions\n",
    "\n",
    "* **mutually exclusive** – any combination of predictors only belongs to a single region, $R_j$\n",
    "\n",
    "* **exhaustive** – all combinations of predictors belong a region, $R_j$, regions cover entire feature space (range of the variables being considered)\n",
    "\n",
    "For every observation in a region, $R_j$, we use the same prediction, $\\hat{Y}(R_j)$    \n",
    "\n",
    "For example predict production, $\\hat{Y}$, from porosity, ${X_1}$\n",
    "\n",
    "* given the data within a mD feature space, $X_1,\\ldots,X_m$, find that boundary maximizes the gap between the two categories\n",
    "\n",
    "* new cases are classified based on where they fall relative to this boundary \n",
    "\n",
    "**Compact, Interpretable Model**\n",
    "\n",
    "Since the classification is based on a hierarchy of binary segmentations of the feature space (one feature at a time) the the model can be specified in a intuitive manner as a:\n",
    "\n",
    "* **tree with binary branches**, hence the name decision tree\n",
    "\n",
    "* **set of nested if statements**, for example:\n",
    "\n",
    "```python\n",
    "if porosity > 0.15:\n",
    "    if brittleness < 20:\n",
    "        initial_production = 1000\n",
    "    else:\n",
    "        initial_production = 7000\n",
    "else:\n",
    "    if brittleness < 40:\n",
    "        initial_production = 500\n",
    "    else:\n",
    "        initial_production = 3000\n",
    "```\n",
    "\n",
    "* **predicts with the average of training response features** in each region $\\hat{Y}(R_j)$.   \n",
    "\n",
    "**Proceedure for Tree Construction**\n",
    "\n",
    "The tree is constructed from the top down.  We begin with a sigle region that covers the entire feature space and then proceed with a sequence of splits.\n",
    "\n",
    "* **Scan All Possible Splits** over all regions and over all features.\n",
    "\n",
    "* **Greedy Optimization**  The method proceeds by finding the first segmentation (split) in any feature that minimizes the residual sum of squares of errors over all the training data $y_i$ over all of the regions $j = 1,\\ldots,J$.\n",
    "\n",
    "\\begin{equation}\n",
    "RSS = \\sum^{J}_{j=1} \\sum_{i \\in R_j} (y_i - \\hat{y}_{R_j})^2\n",
    "\\end{equation}\n",
    "\n",
    "* **Stopping Criteria** is typically based on minimum number of training data in each region for a robust estimation and / or minimum reduction in RSS for the next split \n",
    "\n",
    "##### Applications to Subsurface Modeling\n",
    "\n",
    "To provide a wide variety of spatiotemporal  \n",
    "\n",
    "* we work with just 2 predictor features for the example below for easy visualization\n",
    "\n",
    "* flexibility through selection of kernel (hyperparameter) and associated kernel and cost parameters \n",
    "\n",
    "\n",
    "##### Why Cover Decision Trees?\n",
    "\n",
    "They are not the most powerful, cutting edge method in machine learning\n",
    "\n",
    "* but they are likely the most understandable, interpretable  \n",
    "\n",
    "* decision trees are expanded with random forests, bagging and boosting to be cutting edge  \n",
    "\n",
    "\n",
    "#### Getting Started\n",
    "\n",
    "Here's the steps to get setup in Python with the GeostatsPy package:\n",
    "\n",
    "1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/). \n",
    "2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal. \n",
    "3. In the terminal type: pip install geostatspy. \n",
    "4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality. \n",
    "\n",
    "You may want to copy the data file to your working directory. They are available here:\n",
    "\n",
    "* Tabular data - unconv_MV.csv at https://git.io/fjmBH.\n",
    "\n",
    "or you can use the code below to load the data directly from my GitHub [GeoDataSets](https://github.com/GeostatsGuy/GeoDataSets) repository.\n",
    "\n",
    "#### Install Packages\n",
    "\n",
    "We will include the standard packages for DataFrames and ndarrays and add sci-kit-learn (sklearn) for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "supress_warnings = False\n",
    "import os                                               # to set current working directory \n",
    "import sys                                              # supress output to screen for interactive variogram modeling\n",
    "import numpy as np                                      # arrays and matrix math\n",
    "import pandas as pd                                     # DataFrames\n",
    "import matplotlib.pyplot as plt                         # plotting\n",
    "from sklearn.model_selection import train_test_split    # train and test split\n",
    "from sklearn import tree                                # tree program from scikit learn (package for machine learning)\n",
    "from sklearn import metrics                             # measures to check our models\n",
    "from matplotlib.patches import Rectangle                # build a custom legend\n",
    "from matplotlib.ticker import (MultipleLocator, AutoMinorLocator) # control of axes ticks\n",
    "import math                                             # sqrt operator\n",
    "from ipywidgets import interactive                      # widgets and interactivity\n",
    "from ipywidgets import widgets                            \n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import Label\n",
    "from ipywidgets import VBox, HBox\n",
    "cmap = plt.cm.inferno                                   # default color bar, no bias and friendly for color vision defeciency\n",
    "plt.rc('axes', axisbelow=True)                          # grid behind plotting elements\n",
    "if supress_warnings == True:\n",
    "    import warnings                                     # supress any warnings for this demonstration\n",
    "    warnings.filterwarnings('ignore')                       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.  \n",
    "\n",
    "#### Declare Functions\n",
    "\n",
    "These are functions that I wrote (with one exception) to diagnose and visualize decision tree models.\n",
    "\n",
    "* **extract_rules** - recursive function to get the rules for each devision tree region as a set of conditional statements. Note there is some repeat. I found this on Stack Overflow (I think paulkernfeld was the author).\n",
    "\n",
    "* **plot_decision_tree_regions** - inspired by rstats' tree package, this function takes the extract_rules output to add the tree prediction regions to a plot. Note, only works with 2 predictor features.\n",
    "\n",
    "* **visualize_tree_model** - plot the data in feature space and a grid of the tree predictions. Note, only works for 2 predictor features.\n",
    "\n",
    "* **check_tree_model** - cross validation plot with prediction vs. true for testing and training data. Shows the region values and conditional statistics for the true values in each region (P10, mean and P90)\n",
    "\n",
    "* **tree_tuning** - vary the tree complexity via the number of regions (a.k.a. max_leaf_nodes) hyperparameter and plot the mean square error (MSE) training and testing along with the currently displayed model to visually hyperparameter tune.\n",
    "\n",
    "I'm stoked about these functions. They provide a great improvement in diagnosing and visualizing decision tree models in predictive machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_rules(tree_model, feature_names):           # recursive method to extract rules, from paulkernfeld Stack Overflow (?)\n",
    "    rules = []\n",
    "    def traverse(node, depth, prev_rule):\n",
    "        if tree_model.tree_.children_left[node] == -1:  # Leaf node\n",
    "            class_label = np.argmax(tree_model.tree_.value[node])\n",
    "            rule = f\"{prev_rule} => Class {class_label}\"\n",
    "            rules.append(rule)\n",
    "        else:  # Split node\n",
    "            feature = feature_names[tree_model.tree_.feature[node]]\n",
    "            threshold = tree_model.tree_.threshold[node]\n",
    "            left_child = tree_model.tree_.children_left[node]\n",
    "            right_child = tree_model.tree_.children_right[node]\n",
    "\n",
    "            # Recursively traverse left and right subtrees\n",
    "            traverse(left_child, depth + 1, f\"{prev_rule} & {feature} <= {threshold}\")\n",
    "            traverse(right_child, depth + 1, f\"{prev_rule} & {feature} > {threshold}\")\n",
    "    traverse(0, 0, \"Root\")\n",
    "    return rules\n",
    "\n",
    "def plot_decision_tree_regions(tree_model, feature_names,X_min,X_max):\n",
    "    rules = extract_rules(tree_model, feature_names)\n",
    "    for irule, ____ in enumerate(rules):\n",
    "        rule = rules[irule].split()[2:]\n",
    "        X_min = Xmin[0]; X_max = Xmax[0]; Y_min = Xmin[1]; Y_max = Xmax[1];\n",
    "        index = [i for i,val in enumerate(rule) if val==feature_names[0]]\n",
    "        for i in index:\n",
    "            if rule[i+1] == '<=':\n",
    "                X_max = min(float(rule[i+2]),X_max)\n",
    "            else:\n",
    "                X_min = max(float(rule[i+2]),X_min)\n",
    "        index = [i for i,val in enumerate(rule) if val==feature_names[1]]\n",
    "        for i in index:\n",
    "            if rule[i+1] == '<=':\n",
    "                Y_max = min(float(rule[i+2]),Y_max)\n",
    "            else:\n",
    "                Y_min = max(float(rule[i+2]),Y_min) \n",
    "        plt.gca().add_patch(plt.Rectangle((X_min,Y_min),X_max-X_min,Y_max-Y_min, lw=2,ec='black',fc=\"none\"))\n",
    "        cx = (X_min + X_max)*0.5; cy = (Y_min + Y_max)*0.5; loc = np.array((cx,cy)).reshape(1, -1)\n",
    "        plt.annotate(text = str(np.round(tree_model.predict(loc)[0],2)),xy=(cx,cy),ha='center',weight='bold',c='white',zorder=100)\n",
    "\n",
    "def visualize_tree_model(model,X1_train,X1_test,X2_train,X2_test,Xmin,Xmax,y_train,y_test,ymin,ymax,title,Xname,yname,Xlabel,ylabel):# plots the data points and the decision tree prediction \n",
    "    cmap = plt.cm.inferno\n",
    "    X1plot_step = (Xmax[0] - Xmin[0])/300.0; X2plot_step = -1*(Xmax[1] - Xmin[1])/300.0 # resolution of the model visualization\n",
    "    XX1, XX2 = np.meshgrid(np.arange(Xmin[0], Xmax[0], X1plot_step), # set up the mesh\n",
    "                     np.arange(Xmax[1], Xmin[1], X2plot_step))\n",
    "    y_hat = model.predict(np.c_[XX1.ravel(), XX2.ravel()]) # predict with our trained model over the mesh\n",
    "    y_hat = y_hat.reshape(XX1.shape)\n",
    "\n",
    "    plt.imshow(y_hat,interpolation=None, aspect=\"auto\", extent=[Xmin[0],Xmax[0],Xmin[1],Xmax[1]], \n",
    "        vmin=ymin,vmax=ymax,alpha = 0.2,cmap=cmap,zorder=1)\n",
    "    sp = plt.scatter(X1_train,X2_train,s=None, c=y_train, marker='o', cmap=cmap, \n",
    "        norm=None, vmin=ymin, vmax=ymax, alpha=0.6, linewidths=0.3, edgecolors=\"black\", label = 'Train',zorder=10)\n",
    "    plt.scatter(X1_test,X2_test,s=None, c=y_test, marker='s', cmap=cmap, \n",
    "        norm=None, vmin=ymin, vmax=ymax, alpha=0.3, linewidths=0.3, edgecolors=\"black\", label = 'Test',zorder=10)\n",
    "\n",
    "    plot_decision_tree_regions(model,Xname,Xmin,Xmax)\n",
    "    plt.title(title); plt.xlabel(Xlabel[0]); plt.ylabel(Xlabel[1])\n",
    "    plt.xlim([Xmin[0],Xmax[0]]); plt.ylim([Xmin[1],Xmax[1]])\n",
    "    cbar = plt.colorbar(sp, orientation = 'vertical')   # add the color bar\n",
    "    cbar.set_label(ylabel, rotation=270, labelpad=20)\n",
    "    return y_hat\n",
    "\n",
    "def check_tree_model(model,X1_train,X1_test,X2_train,X2_test,Xmin,Xmax,y_train,y_test,ymin,ymax,title):    # plots the estimated vs. the actual  \n",
    "    y_hat_train = model.predict(np.c_[X1_train,X2_train]); y_hat_test = model.predict(np.c_[X1_test,X2_test])\n",
    "\n",
    "    df_cross = pd.DataFrame(np.c_[y_test,y_hat_test],columns=['y_test','y_hat_test'])\n",
    "    df_cross_train = pd.DataFrame(np.c_[y_train,y_hat_train],columns=['y_train','y_hat_train'])\n",
    "\n",
    "    plt.scatter(y_train,y_hat_train,s=15, c='blue',marker='o', cmap=None, norm=None, vmin=None, vmax=None, alpha=0.7, \n",
    "                linewidths=0.3, edgecolors=\"black\",label='Train',zorder=10)\n",
    "    plt.scatter(y_test,y_hat_test,s=15, c='red',marker='s', cmap=None, norm=None, vmin=None, vmax=None, alpha=0.7, \n",
    "                linewidths=0.3, edgecolors=\"black\",label='Test',zorder=10)\n",
    "\n",
    "    unique_y_hat_all = set(np.concatenate([y_hat_test,y_hat_train]))\n",
    "    for y_hat in unique_y_hat_all:\n",
    "        plt.plot([ymin,ymax],[y_hat,y_hat],c='black',alpha=0.2,ls='--',zorder=1)\n",
    "        \n",
    "    unique_y_hat_test = set(y_hat_test)\n",
    "    for y_hat in unique_y_hat_test:\n",
    "        #plt.plot([ymin,ymax],[y_hat,y_hat],c='black',alpha=0.3,ls='--',zorder=1)\n",
    "        cond_mean_y_hat = df_cross.loc[df_cross['y_hat_test'] == y_hat, 'y_test'].mean()\n",
    "        cond_P75_y_hat = df_cross.loc[df_cross['y_hat_test'] == y_hat, 'y_test'].quantile(0.75)\n",
    "        cond_P25_y_hat = df_cross.loc[df_cross['y_hat_test'] == y_hat, 'y_test'].quantile(0.25)\n",
    "        plt.scatter(cond_mean_y_hat,y_hat-0.02*(ymax-ymin),color='red',edgecolor='black',s=60,marker='^',zorder=100)\n",
    "        plt.plot([cond_P25_y_hat,cond_P75_y_hat],[y_hat-0.025*(ymax-ymin),y_hat-0.025*(ymax-ymin)],c='black',lw=0.7)\n",
    "        plt.plot([cond_P25_y_hat,cond_P25_y_hat],[y_hat-0.032*(ymax-ymin),y_hat-0.018*(ymax-ymin)],c='black',lw=0.7)\n",
    "        plt.plot([cond_P75_y_hat,cond_P75_y_hat],[y_hat-0.032*(ymax-ymin),y_hat-0.018*(ymax-ymin)],c='black',lw=0.7)\n",
    "        \n",
    "    unique_y_hat_train = set(y_hat_train)\n",
    "    for y_hat in unique_y_hat_train:\n",
    "        #plt.plot([ymin,ymax],[y_hat,y_hat],c='black',alpha=0.3,ls='--',zorder=1)\n",
    "        cond_mean_y_hat = df_cross_train.loc[df_cross_train['y_hat_train'] == y_hat, 'y_train'].mean()\n",
    "        cond_P75_y_hat = df_cross_train.loc[df_cross_train['y_hat_train'] == y_hat, 'y_train'].quantile(0.75)\n",
    "        cond_P25_y_hat = df_cross_train.loc[df_cross_train['y_hat_train'] == y_hat, 'y_train'].quantile(0.25)\n",
    "        plt.scatter(cond_mean_y_hat,y_hat+0.02*(ymax-ymin),color='blue',edgecolor='black',s=60,marker='v',zorder=100)\n",
    "        plt.plot([cond_P25_y_hat,cond_P75_y_hat],[y_hat+0.025*(ymax-ymin),y_hat+0.025*(ymax-ymin)],c='black',lw=0.7)\n",
    "        plt.plot([cond_P25_y_hat,cond_P25_y_hat],[y_hat+0.032*(ymax-ymin),y_hat+0.018*(ymax-ymin)],c='black',lw=0.7)\n",
    "        plt.plot([cond_P75_y_hat,cond_P75_y_hat],[y_hat+0.032*(ymax-ymin),y_hat+0.018*(ymax-ymin)],c='black',lw=0.7)\n",
    "        \n",
    "    plt.title(title); plt.xlabel('Actual Production (MCFPD)'); plt.ylabel('Estimated Production (MCFPD)')\n",
    "    plt.xlim([ymin,ymax]); plt.ylim([ymin,ymax]); plt.legend(loc='upper left')\n",
    "    plt.gca().grid(True, which='major',linewidth = 1.0); plt.gca().grid(True, which='minor',linewidth = 0.2) # add y grids\n",
    "    plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)\n",
    "    plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks\n",
    "\n",
    "    plt.arrow(ymin,ymin,ymax,ymax,width=0.02,color='black',head_length=0.0,head_width=0.0)\n",
    "    MSE_train = metrics.mean_squared_error(y_train,y_hat_train); MSE_test = metrics.mean_squared_error(y_test,y_hat_test)\n",
    "    Var_Explained = metrics.explained_variance_score(y_test,y_hat_test)\n",
    "    cor = math.sqrt(metrics.r2_score(y_test,y_hat_test))\n",
    "    plt.gca().add_patch(plt.Rectangle((ymin+0.6*(ymax-ymin),ymin+0.1*(ymax-ymin)),0.40*(ymax-ymin),0.12*(ymax-ymin),\n",
    "        lw=0.5,ec='black',fc=\"white\",zorder=100))\n",
    "    plt.annotate('MSE Testing: ' + str(np.round(MSE_test,2)),(ymin+0.62*(ymax-ymin),ymin+0.18*(ymax-ymin)),zorder=1000)\n",
    "    plt.annotate('MSE Training: ' + str(np.round(MSE_train,2)),(ymin+0.62*(ymax-ymin),ymin+0.12*(ymax-ymin)),zorder=1000)\n",
    "    \n",
    "def tree_tuning(node_max,cnode,X1_train,X1_test,X2_train,X2_test,Xmin,Xmax,y_train,y_test,ymin,ymax,title,seed):\n",
    "    MSE_test_mat = np.zeros(node_max-1); MSE_train_mat = np.zeros(node_max-1);\n",
    "    \n",
    "    for imax_leaf_node, max_leaf_node in enumerate(range(2,node_max+1)):\n",
    "        np.random.seed(seed = seed)\n",
    "        tree_temp = tree.DecisionTreeRegressor(max_leaf_nodes = max_leaf_node)\n",
    "        tree_temp = tree_temp.fit(X_train.values, y_train.values)\n",
    "        y_hat_train = tree_temp.predict(np.c_[X1_train,X2_train]); y_hat_test = tree_temp.predict(np.c_[X1_test,X2_test])  \n",
    "        MSE_train_mat[imax_leaf_node] = metrics.mean_squared_error(y_train,y_hat_train)\n",
    "        MSE_test_mat[imax_leaf_node] = metrics.mean_squared_error(y_test,y_hat_test)\n",
    "        if max_leaf_node == cnode:\n",
    "            plt.scatter(cnode,MSE_train_mat[imax_leaf_node],color='blue',edgecolor='black',s=20,marker='o',zorder=1000)\n",
    "            plt.scatter(cnode,MSE_test_mat[imax_leaf_node],color='red',edgecolor='black',s=20,marker='o',zorder=1000)\n",
    "    maxcheck = max(np.max(MSE_train_mat),np.max(MSE_test_mat))\n",
    "    \n",
    "    plt.vlines(cnode,0,maxcheck,color='black',ls='--',lw=1,zorder=1) \n",
    "    plt.plot(range(2,node_max+1),MSE_train_mat,color='blue',zorder=100,label='Train')\n",
    "    plt.plot(range(2,node_max+1),MSE_test_mat,color='red',zorder=100,label='Test')\n",
    "    \n",
    "    plt.title(title); plt.xlabel('Maximum Number of Leaf Nodes'); plt.ylabel('Means Square Error')\n",
    "    plt.xlim([0,node_max]); plt.ylim([0,maxcheck]); plt.legend(loc='upper right')\n",
    "    plt.gca().grid(True, which='major',linewidth = 1.0); plt.gca().grid(True, which='minor',linewidth = 0.2) # add y grids\n",
    "    plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)\n",
    "    plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Data\n",
    "\n",
    "Let's load the provided multivariate, spatial dataset [unconv_MV.csv](https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/unconv_MV.csv) available in my GeoDataSet repo. It is a comma delimited file with: \n",
    "\n",
    "* well index (integer)\n",
    "* porosity (fraction)\n",
    "* permeability ($mD$)\n",
    "* acoustic impedance ($\\frac{kg}{m^3} \\cdot \\frac{m}{s} \\cdot 10^6$). \n",
    "* total organic carbon (%)\n",
    "* vitrinite reflectance (%)\n",
    "* initial gass production (90 day average) (MCFPD)\n",
    "\n",
    "We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it to make sure it loaded correctly.\n",
    "\n",
    "**Python Tip: using functions from a package** just type the label for the package that we declared at the beginning:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "```\n",
    "\n",
    "so we can access the pandas function 'read_csv' with the command: \n",
    "\n",
    "```python\n",
    "pd.read_csv()\n",
    "```\n",
    "\n",
    "but read csv has required input parameters. The essential one is the name of the file. For our circumstance all the other default parameters are fine. If you want to see all the possible parameters for this function, just go to the docs [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).  \n",
    "\n",
    "* The docs are always helpful\n",
    "* There is often a lot of flexibility for Python functions, possible through using various inputs parameters\n",
    "\n",
    "also, the program has an output, a pandas DataFrame loaded from the data.  So we have to specficy the name / variable representing that new object.\n",
    "\n",
    "```python\n",
    "df = pd.read_csv(\"unconv_MV.csv\")  \n",
    "```\n",
    "\n",
    "Let's run this command to load the data and then this command to extract a random subset of the data.\n",
    "\n",
    "```python\n",
    "df = df.sample(frac=.30, random_state = 73073); \n",
    "df = df.reset_index()\n",
    "```\n",
    "\n",
    "#### Feature Engineering\n",
    "\n",
    "Let's make some changes to the data to improve the workflow:\n",
    "\n",
    "* **Select the predictor features (x2) and the response feature (x1)**, make sure the metadata is also consistent.\n",
    "* **Metadata** encoding such as the units, labels and display ranges for each feature.\n",
    "* **Reduce the number of data** for ease of visualization (hard to see if too many points on our plots).\n",
    "* **Train and test data split** to demonstrate and visualize simple hyperparameter tuning.\n",
    "* **Add random noise to the data** to demonstrate model overfit. The original data is error free and does not readily demonstrate overfit.\n",
    "\n",
    "Given this is properly set, one should be able to use any dataset and features for this demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 67048 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "add_error = True                                        # add random error to the response feature\n",
    "std_error = 700; seed = 71071\n",
    "\n",
    "yname = 'Production'; Xname = ['Por','Brittle']         # specify the predictor features (x2) and response feature (x1)\n",
    "Xmin = [5.0,0.0]; Xmax = [25.0,100.0]                   # set minumums and maximums for visualization \n",
    "ymin = 0.0; ymax = 10000.0\n",
    "Xlabel = ['Porosity','Brittleness']; ylabel = 'Production' # specify the feature labels for plotting\n",
    "Xunit = ['%','fraction']; yunit = 'MCFPD'\n",
    "Xlabelunit = [Xlabel[0] + ' (' + Xunit[0] + ')',Xlabel[1] + ' (' + Xunit[1] + ')']\n",
    "ylabelunit = ylabel + ' (' + yunit + ')'\n",
    "\n",
    "#df = pd.read_csv(\"unconv_MV.csv\")                      # load the data from local current directory\n",
    "df = pd.read_csv(r\"https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/unconv_MV.csv\") # load the data from my github repo\n",
    "df = df.sample(frac=.30, random_state = 73073); df = df.reset_index() # extract 30% random to reduce the number of data\n",
    "\n",
    "if add_error == True:                                   # method to add error\n",
    "    np.random.seed(seed=seed)                           # set random number seed\n",
    "    df[yname] = df[yname] + np.random.normal(loc = 0.0,scale=std_error,size=len(df)) # add noise\n",
    "    values = df._get_numeric_data(); values[values < 0] = 0  # set negative to 0 in a shallow copy ndarray\n",
    "    \n",
    "y = pd.DataFrame(df[yname])                             # extract selected features as X and y DataFrames\n",
    "X = df[Xname]\n",
    "df = pd.concat([X,y],axis=1)                            # make one DataFrame with both X and y (remove all other features)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.25,random_state=73073) # train and test split\n",
    "# y_train = pd.DataFrame({yname:y_train.values}); y_test = pd.DataFrame({yname:y_test.values}) # optional to ensure response is a DataFrame\n",
    "\n",
    "plt.subplot(121)                                        # visualize the data\n",
    "im = plt.scatter(X_train[Xname[0]],X_train[Xname[1]],s=None, c=y_train[yname], marker='o', cmap=cmap, \n",
    "    norm=None, vmin=ymin, vmax=ymax, alpha=0.8, linewidths=0.3, edgecolors=\"black\", label = 'Train')\n",
    "plt.scatter(X_test[Xname[0]],X_test[Xname[1]],s=None, c=y_test[yname], marker='s', cmap=cmap, \n",
    "    norm=None, vmin=ymin, vmax=ymax, alpha=0.5, linewidths=0.3, edgecolors=\"black\", label = 'Test')\n",
    "plt.title('Training ' + ylabel + ' vs. ' + Xlabel[1] + ' and ' + Xlabel[0]); \n",
    "plt.xlabel(Xlabel[0] + ' (' + Xunit[0] + ')'); plt.ylabel(Xlabel[1] + ' (' + Xunit[1] + ')')\n",
    "plt.xlim(Xmin[0],Xmax[0]); plt.ylim(Xmin[1],Xmax[1]); plt.legend(loc = 'upper right'); plt.grid(True)\n",
    "cbar = plt.colorbar(im, orientation = 'vertical')\n",
    "cbar.set_label(ylabel + ' (' + yunit + ')', rotation=270, labelpad=20)\n",
    "\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interactive Decision Tree Method\n",
    "\n",
    "The following code includes a dashboard with widgets linked to plots to run a decision tree predictive machine learning model and to provide diagnostics.\n",
    "\n",
    "* visualization of the decision tree regions with the estimate in each region annotated\n",
    "* cross validation plot with conditional mean, P10 and P90 of the truth for each possible estimate\n",
    "* the MSE train and test curves for hyperparameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "l = widgets.Text(value='                                                Machine Learning Decision Tree Demo, Prof. Michael Pyrcz, The University of Texas at Austin',\n",
    "        layout=Layout(width='970px', height='30px'))\n",
    "\n",
    "max_leaf_nodes = widgets.IntSlider(min=2, max = 150, value=2, step = 1, description = r'$n_{leaf}$',orientation='horizontal', \n",
    "        style = {'description_width': 'initial'},layout=Layout(width='900px', height='30px'),continuous_update=False)\n",
    "\n",
    "ui_summary = widgets.HBox([max_leaf_nodes,],)\n",
    "ui2_summary = widgets.VBox([l,ui_summary],)\n",
    "\n",
    "def run_plot_summary(max_leaf_nodes):\n",
    "    global tree_model                                   # make the current tree model available for additional visualizations\n",
    "    tree_model = tree.DecisionTreeRegressor(max_leaf_nodes = max_leaf_nodes) # instantiate and train tree model\n",
    "    tree_model = tree_model.fit(X_train.values, y_train.values)\n",
    "    \n",
    "    plt.subplot(131)                                    # visualize, data, and decision tree regions and predictions\n",
    "    visualize_tree_model(tree_model,X_train[Xname[0]],X_test[Xname[0]],X_train[Xname[1]],X_test[Xname[1]],Xmin,Xmax,\n",
    "                    y_train[yname],y_test[yname],ymin,ymax,'Decision Tree Model',Xname,yname,Xlabelunit,ylabelunit) # plots the data points and the decision tree prediction \n",
    "    \n",
    "    plt.subplot(132)                                    # cross validation with conditional statistics plot\n",
    "    check_tree_model(tree_model,X_train[Xname[0]],X_test[Xname[0]],X_train[Xname[1]],X_test[Xname[1]],Xmin,Xmax,\n",
    "                    y_train[yname],y_test[yname],ymin,ymax,'Decision Tree Model',)\n",
    "    \n",
    "    plt.subplot(133)                                    # hyperparameter tuning plot with current result plotted\n",
    "    tree_tuning(100,max_leaf_nodes,X_train[Xname[0]],X_test[Xname[0]],X_train[Xname[1]],X_test[Xname[1]],Xmin,Xmax,\n",
    "                    y_train[yname],y_test[yname],ymin,ymax,'Decision Tree Model Error',seed)\n",
    "    \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.3, hspace=0.2); plt.show()\n",
    "    \n",
    "    return tree_model\n",
    "    \n",
    "# connect the function to make the samples and plot to the widgets    \n",
    "interactive_plot_summary = widgets.interactive_output(run_plot_summary, {'max_leaf_nodes':max_leaf_nodes,})\n",
    "interactive_plot_summary.clear_output(wait = True)                  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Decision Tree Demonstration\n",
    "\n",
    "* select maximum number of leaf nodes, $n_{leaf}$, and observe the decision tree model, cross validation plot and entire training and testing error curves.\n",
    "\n",
    "#### Michael Pyrcz, Professor, University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)\n",
    "\n",
    "### The Inputs\n",
    "\n",
    "Change the complexity of the decision tree model\n",
    "\n",
    "* $n_{leaf}$: the maximum number of leaf nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89da3fe981ee405499a8dbb2ba5100b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='                                                Machine Learning Decision Tree Demo…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24973261c6124fc09e5611c16a100b00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '<Figure size 432x288 with 4 Axes>', 'i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui2_summary, interactive_plot_summary)                           # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Additional Tree Model Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35652 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tree.plot_tree(tree_model,label='none',impurity=False)\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comments\n",
    "\n",
    "This was an interactive demonstration of predictive machine learning with decision trees. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy. \n",
    "  \n",
    "#### The Author:\n",
    "\n",
    "### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)  \n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
