{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction: Looking at Data to Correct Cognitive Biases\n",
"\n",
"[Medium article: \"How to Overcome Biases with Data\"](https://medium.com/p/692c8c35f4a5)\n",
"\n",
"In this notebook, we will investigate a simple problem: is the smoking rate higher in the metro area of my hometown (Peoria, Il) or in Boston. The purpose is to see if I can combat my cognitive biases with data and explore the data to determine why I made a incorrect conclusion.\n",
"\n",
"# City Level Data\n",
"\n",
"https://nccd.cdc.gov/500_Cities/rdPage.aspx?rdReport=DPH_500_Cities.ComparisonReport&Locations=1759000,2507000&rdRequestForwarding=Form\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:25.874443Z",
"start_time": "2018-11-02T02:22:25.695890Z"
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:25.878214Z",
"start_time": "2018-11-02T02:22:25.875892Z"
}
},
"outputs": [],
"source": [
"# Boston\n",
"b_p = 0.164\n",
"b_ci = [0.161, 0.166]\n",
"b_adults = 472582\n",
"b_smokers = b_p * b_adults\n",
"\n",
"# Peoria\n",
"p_p = 0.199\n",
"p_ci = [0.194, 0.203]\n",
"p_adults = 83925\n",
"p_smokers = p_p * p_adults"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Difference Between Two Proportions\n",
"\n",
"This is assuming 10% of adults in each city surveyed. \n",
"\n",
"Relevant Data:\n",
"\n",
"* Boston number of adults = 472582: https://www.bumc.bu.edu/inspir/files/HTML/Boston%20Population%20and%20Demographics.htm\n",
"* Peoria number of adults = 83925: https://www.infoplease.com/us/illinois/demographic-statistics-61\n",
"* Boston smoking rate = 0.164 (from first source in image)\n",
"* Peoria smoking rate = 0.199(from first source in image)\n",
"\n",
"z-score for 95% CI: http://www.stat.yale.edu/Courses/1997-98/101/confint.htm\n",
"\n",
"Difference between two proportions formula: https://onlinecourses.science.psu.edu/stat100/node/57/"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:25.884108Z",
"start_time": "2018-11-02T02:22:25.879415Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The 95% confidence interval for difference between smoking rates is from 2.5829% to 4.4171%\n"
]
}
],
"source": [
"# Standard errors individual\n",
"b_se = np.sqrt((b_p * (1 - b_p)) / (0.1 * b_adults))\n",
"p_se = np.sqrt((p_p * (1 - p_p)) / (0.1 * p_adults))\n",
"\n",
"# Standard error combined\n",
"se_diff = np.sqrt(np.square(b_se) + np.square(p_se))\n",
"\n",
"z_95 = 1.96\n",
"\n",
"# Upper confidence bound for difference (two-tailed)\n",
"uci = (p_p - b_p) + z_95 * se_diff\n",
"# Lower confidence bound\n",
"lci = (p_p - b_p) - z_95 * se_diff\n",
"\n",
"print(f'The 95% confidence interval for difference between smoking rates is from {round(100 * lci, 4)}% to {round(100 * uci, 4)}%')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This indicates the difference is significant. We can also tell this from the non-overlapping individual confidence intervals."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Distributions\n",
"\n",
"If we assume the smoking rate proportion is normally distributed then we can construct a normal distribution and sample from it. Then we can visually show the data in boxplots."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:25.902707Z",
"start_time": "2018-11-02T02:22:25.885493Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Standard deviation for Boston: 0.001020408163265307\n",
"Standard deviation for Peoria: 0.002040816326530614\n"
]
}
],
"source": [
"z_95 = 1.96\n",
"\n",
"b_sigma = ((b_ci[1] - b_p) / z_95) * 1\n",
"p_sigma = ((p_ci[1] - p_p) / z_95) * 1\n",
"\n",
"print(f'Standard deviation for Boston: {b_sigma}')\n",
"print(f'Standard deviation for Peoria: {p_sigma}')\n",
"\n",
"# Sample distributions \n",
"p_dist = np.random.normal(loc = p_p, scale = p_sigma, size = 100000)\n",
"b_dist = np.random.normal(loc = b_p, scale = b_sigma, size = 100000)\n",
"# Names\n",
"b = np.repeat('boston', repeats = len(b_dist))\n",
"p = np.repeat('peoria', repeats = len(p_dist))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confidence interval for Boston."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:26.108357Z",
"start_time": "2018-11-02T02:22:25.904047Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"(0.16200003675046934, 0.16599996324953067)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from scipy import stats\n",
"b_ci = stats.norm.interval(0.95, loc = b_p, scale = b_sigma)\n",
"b_ci"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confidence interval for Peoria"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:26.113098Z",
"start_time": "2018-11-02T02:22:26.109723Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"(0.19500007350093868, 0.20299992649906135)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"p_ci = stats.norm.interval(0.95, loc = p_p, scale = p_sigma)\n",
"p_ci"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The confidence intervals from our simulated data are basically the same as those from the survey data. The survey data does have uneven confidence intervals though. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2018-11-02T02:22:26.377082Z",
"start_time": "2018-11-02T02:22:26.114286Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" proportion | \n",
"
\n",
" \n",
" | \n",
" count | \n",
" mean | \n",
" std | \n",
" min | \n",
" 25% | \n",
" 50% | \n",
" 75% | \n",
" max | \n",
"
\n",
" \n",
" | city | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | boston | \n",
" 100000.0 | \n",
" 0.164003 | \n",
" 0.001018 | \n",
" 0.159767 | \n",
" 0.163316 | \n",
" 0.164006 | \n",
" 0.164690 | \n",
" 0.168284 | \n",
"
\n",
" \n",
" | peoria | \n",
" 100000.0 | \n",
" 0.199006 | \n",
" 0.002042 | \n",
" 0.190117 | \n",
" 0.197627 | \n",
" 0.199007 | \n",
" 0.200389 | \n",
" 0.209150 | \n",
"
\n",
" \n",
"
\n",
"