{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction Simpson's Paradox\n",
    "\n",
    "Simpson's paradox occurs when trends that are present when data is separated into groups reverse when the data is aggregated. In this notebook, we take a look at three simple examples of Simpson's Paradox both quantitatively and visually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Medicine A vs Medicine B\n",
    "\n",
    "In this example, X is better for both males and females, but performs worse overall! This is due to different sample sizes of males and females receiving each treatment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X Better</th>\n",
       "      <th>X Treated</th>\n",
       "      <th>Y Better</th>\n",
       "      <th>Y Treated</th>\n",
       "      <th>X Percent</th>\n",
       "      <th>Y Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>M</th>\n",
       "      <td>125</td>\n",
       "      <td>610</td>\n",
       "      <td>29</td>\n",
       "      <td>150</td>\n",
       "      <td>0.204918</td>\n",
       "      <td>0.193333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <td>22</td>\n",
       "      <td>75</td>\n",
       "      <td>60</td>\n",
       "      <td>240</td>\n",
       "      <td>0.293333</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   X Better  X Treated  Y Better  Y Treated  X Percent  Y Percent\n",
       "M       125        610        29        150   0.204918   0.193333\n",
       "F        22         75        60        240   0.293333   0.250000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({'X Better': [125, 22], 'X Treated': [610, 75], 'Y Better': [29, 60], 'Y Treated': [150, 240]}, index = [\"M\", \"F\"])\n",
    "data['X Percent'] = data['X Better'] / data['X Treated']\n",
    "data['Y Percent'] = data['Y Better'] / data['Y Treated']\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X Better</th>\n",
       "      <th>X Treated</th>\n",
       "      <th>Y Better</th>\n",
       "      <th>Y Treated</th>\n",
       "      <th>X Percent</th>\n",
       "      <th>Y Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>M</th>\n",
       "      <td>125.0</td>\n",
       "      <td>610.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.204918</td>\n",
       "      <td>0.193333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <td>22.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>240.0</td>\n",
       "      <td>0.293333</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>combined</th>\n",
       "      <td>147.0</td>\n",
       "      <td>685.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>390.0</td>\n",
       "      <td>0.214599</td>\n",
       "      <td>0.228205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          X Better  X Treated  Y Better  Y Treated  X Percent  Y Percent\n",
       "M            125.0      610.0      29.0      150.0   0.204918   0.193333\n",
       "F             22.0       75.0      60.0      240.0   0.293333   0.250000\n",
       "combined     147.0      685.0      89.0      390.0   0.214599   0.228205"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc['combined', :4] = list(data.iloc[:, :4].sum())\n",
    "data.loc['combined', 'X Percent'] = data.loc['combined', 'X Better'] / data.loc['combined', 'X Treated']\n",
    "data.loc['combined', 'Y Percent'] = data.loc['combined', 'Y Better'] / data.loc['combined', 'Y Treated']\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Restaurant Reviews\n",
    "\n",
    "In this example, Carlo's Restaurant is recommended by a higher percentage of both males and females, but is recommended by a lower overall percentage of respondents. This again occurs because of mismatched sample sizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Recommend Sophia's</th>\n",
       "      <th>Sophia's Total</th>\n",
       "      <th>Recommend Carlo's</th>\n",
       "      <th>Carlo's Total</th>\n",
       "      <th>Sophia's Percent</th>\n",
       "      <th>Carlo's Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>M</th>\n",
       "      <td>50</td>\n",
       "      <td>150</td>\n",
       "      <td>180</td>\n",
       "      <td>360</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <td>200</td>\n",
       "      <td>250</td>\n",
       "      <td>36</td>\n",
       "      <td>40</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Recommend Sophia's  Sophia's Total  Recommend Carlo's  Carlo's Total  \\\n",
       "M                  50             150                180            360   \n",
       "F                 200             250                 36             40   \n",
       "\n",
       "   Sophia's Percent  Carlo's Percent  \n",
       "M          0.333333              0.5  \n",
       "F          0.800000              0.9  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({\"Recommend Sophia's\": [50, 200], \"Sophia's Total\": [150, 250], \"Recommend Carlo's\": [180, 36], \"Carlo's Total\": [360, 40]}, index = [\"M\", \"F\"])\n",
    "data[\"Sophia\\'s Percent\"] = data[\"Recommend Sophia's\"] / data[\"Sophia's Total\"]\n",
    "data[\"Carlo's Percent\"] = data[\"Recommend Carlo's\"] / data[\"Carlo's Total\"]\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Recommend Sophia's</th>\n",
       "      <th>Sophia's Total</th>\n",
       "      <th>Recommend Carlo's</th>\n",
       "      <th>Carlo's Total</th>\n",
       "      <th>Sophia's Percent</th>\n",
       "      <th>Carlo's Percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>M</th>\n",
       "      <td>50.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <td>200.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>combined</th>\n",
       "      <td>250.0</td>\n",
       "      <td>400.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>400.0</td>\n",
       "      <td>0.625000</td>\n",
       "      <td>0.54</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Recommend Sophia's  Sophia's Total  Recommend Carlo's  \\\n",
       "M                       50.0           150.0              180.0   \n",
       "F                      200.0           250.0               36.0   \n",
       "combined               250.0           400.0              216.0   \n",
       "\n",
       "          Carlo's Total  Sophia's Percent  Carlo's Percent  \n",
       "M                 360.0          0.333333             0.50  \n",
       "F                  40.0          0.800000             0.90  \n",
       "combined          400.0          0.625000             0.54  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc['combined', :4] = list(data.iloc[:, :4].sum())\n",
    "data.loc['combined', \"Sophia's Percent\"] = data.loc['combined', \"Recommend Sophia's\"] / data.loc['combined', \"Sophia's Total\"]\n",
    "data.loc['combined', \"Carlo's Percent\"] = data.loc['combined', \"Recommend Carlo's\"] / data.loc['combined', \"Carlo's Total\"]\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Correlation Reversal: Hours of Exercise vs Chance of Developing Disease\n",
    "\n",
    "This is another fictional example showing Simpson's Paradox in the context of a correlation reversal. When looking at the data individually, there is a negative correlation between hours of exercise and chance of developing a disease, but when aggregating the data, the correlation reverses! This is due to the presence of another cause, age, on the chance of developing a disease. In order to determine the effect of exercise on the probability of disease, we need to control for the age of patients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.polynomial.polynomial import polyfit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 95668 characters]\n",
      "text/plain": [
       "<Figure size 1440x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_samples = 100\n",
    "np.random.seed(42)\n",
    "ages = np.random.randint(20, 50, n_samples)\n",
    "hours = np.random.randint(1, 5, n_samples) + np.random.randn(n_samples)\n",
    "p = 12 + 0.5 * ages + -2.1 * hours + np.random.randn(n_samples) *  2\n",
    "under_50 = pd.DataFrame({'age': ages, 'Hours Exercised': hours, 'probability': p})\n",
    "\n",
    "n_samples = 100\n",
    "ages = np.random.randint(50, 85, n_samples)\n",
    "hours = np.random.randint(3, 8, n_samples) + np.random.randn(n_samples) * 0.5\n",
    "p = 40 + 0.32 * ages + -3.2 * hours + np.random.randn(n_samples) \n",
    "over_50 = pd.DataFrame({'age': ages, 'Hours Exercised': hours, 'probability': p})\n",
    "\n",
    "\n",
    "def plot_relationship(data, c, color, ax):\n",
    "    \"\"\"Plot a scatter plot with linear fit\"\"\"\n",
    "    x, y = np.array(data[c]), np.array(data['probability'])\n",
    "    # Linear fit (polynomial of degree 1)\n",
    "    b, m = polyfit(x, y, 1)\n",
    "    # Plot scatterplot\n",
    "    data.plot(x = c, y = 'probability', c = color, \n",
    "              style = 'o', legend = None, ax = ax, ms = 10)\n",
    "    # Plot linear fit\n",
    "    ax.plot(x, m * x + b, '-', color = 'k');\n",
    "    if color == '#d9d142':\n",
    "        plt.title(f'Probability vs {c.capitalize()} over 50')\n",
    "    elif color == '#04c5ff':\n",
    "        plt.title(f'Probability vs {c.capitalize()} under 50')\n",
    "    else:\n",
    "        plt.title(f'Probability vs {c.capitalize()} Combined')\n",
    "    corr_coef = np.corrcoef(x, y)[0][1]\n",
    "    ax = plt.gca()\n",
    "    plt.ylabel('Probability'); \n",
    "    plt.text(0.2, 0.75, r'$\\rho$ = ' + f'{round(corr_coef, 2)}', fontsize = 28, color = 'k', \n",
    "             transform=ax.transAxes)\n",
    "\n",
    "    \n",
    "plt.figure(figsize = (20, 8))\n",
    "ax = plt.subplot(1, 2, 1)\n",
    "plot_relationship(under_50, 'Hours Exercised', '#04c5ff', ax)\n",
    "ax = plt.subplot(1, 2, 2)\n",
    "plot_relationship(over_50, 'Hours Exercised', '#d9d142', ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54924 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10, 8))\n",
    "combined = pd.concat([under_50, over_50], axis = 0)\n",
    "ax = plt.subplot(1, 1, 1)\n",
    "plot_relationship(combined, 'Hours Exercised', 'r', ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 89136 characters]\n",
      "text/plain": [
       "<Figure size 1440x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (20, 8))\n",
    "ax = plt.subplot(1, 2, 1)\n",
    "plot_relationship(under_50, 'age', '#04c5ff', ax)\n",
    "ax = plt.subplot(1, 2, 2)\n",
    "plot_relationship(over_50, 'age', '#d9d142', ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 55484 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10, 8))\n",
    "ax = plt.subplot(1, 1, 1)\n",
    "plot_relationship(combined, 'age', 'r', ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = [ '#04c5ff' for _ in range(n_samples)]\n",
    "colors.extend(['#d9d142' for _ in range(n_samples)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 85372 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10, 8))\n",
    "\n",
    "plt.scatter(combined['Hours Exercised'],\n",
    "            combined['probability'], c = colors, label = None, s = 60);\n",
    "\n",
    "x_c, y_c = np.array(combined['Hours Exercised']), np.array(combined['probability'])\n",
    "b_c, m_c = polyfit(x_c, y_c, 1)\n",
    "\n",
    "x_u, y_u = np.array(under_50['Hours Exercised']), np.array(under_50['probability'])\n",
    "b_u, m_u = polyfit(x_u, y_u, 1)\n",
    "\n",
    "x_o, y_o = np.array(over_50['Hours Exercised']), np.array(over_50['probability'])\n",
    "b_o, m_o = polyfit(over_50['Hours Exercised'], over_50['probability'], 1)\n",
    "\n",
    "plt.plot(x_u, b_u + m_u * x_u, c =  '#04c5ff', label = 'Under 50 Fit');\n",
    "plt.plot(x_o, b_o + m_o * x_o, c = '#d9d142', label = 'Over 50 Fit');\n",
    "plt.plot(x_c, b_c + m_c * x_c, c = 'r', label = 'Combined Fit');\n",
    "plt.xlabel('Hours Exercised');\n",
    "plt.ylabel('Probability'); plt.title(\"Simpson's Paradox: Correlation Reversal\");\n",
    "plt.legend(prop = {'size': 14});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 84392 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10, 8))\n",
    "\n",
    "plt.scatter(combined['age'],\n",
    "            combined['probability'], c = colors, label = None, s = 60);\n",
    "\n",
    "x_c, y_c = np.array(combined['age']), np.array(combined['probability'])\n",
    "b_c, m_c = polyfit(x_c, y_c, 1)\n",
    "\n",
    "x_u, y_u = np.array(under_50['age']), np.array(under_50['probability'])\n",
    "b_u, m_u = polyfit(x_u, y_u, 1)\n",
    "\n",
    "x_o, y_o = np.array(over_50['age']), np.array(over_50['probability'])\n",
    "b_o, m_o = polyfit(over_50['age'], over_50['probability'], 1)\n",
    "\n",
    "plt.plot(x_u, b_u + m_u * x_u, c =  '#04c5ff', label = 'Under 50 Fit');\n",
    "plt.plot(x_o, b_o + m_o * x_o, c = '#d9d142', label = 'Over 50 Fit');\n",
    "plt.plot(x_c, b_c + m_c * x_c, c = 'r', label = 'Combined Fit');\n",
    "plt.xlabel('Age');\n",
    "plt.ylabel('Probability'); plt.title(\"Simpson's Paradox: Additional Cause\");\n",
    "plt.legend(prop = {'size': 14});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 146424 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "fig = plt.figure(figsize = (10, 8))\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "\n",
    "\n",
    "markers = ['.' for _ in range(100)]\n",
    "markers.extend('o' for _ in range(100))\n",
    "\n",
    "x_c = np.array(combined['Hours Exercised'])\n",
    "y_c = np.array(combined['age'])\n",
    "z_c = np.array(combined['probability'])\n",
    "ax.scatter(x_c, y_c, z_c, c=colors, s = 40, marker='o', alpha = 1.0)\n",
    "\n",
    "ax.set_xlabel('Hours Exercised')\n",
    "ax.set_ylabel('Age')\n",
    "ax.set_zlabel('Probability');\n",
    "ax.set_title('Visualization of Multiple Causes', y = 1.05);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "Simpson's Paradox is an intriguing statistical phenomenon that demonstrates the importance of thinking causally by asking why - determining the data generation process. Once we know how the data is generated, we can look for the other factors we are not being shown and resolve the paradox by either keeping the data separated, or aggregating the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
