{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Part 1 - Basic operations\n",
    "\n",
    "![](img/titanic.jpg)\n",
    "\n",
    "## Titanic dataset\n",
    "\n",
    "Dataset source: https://www.kaggle.com/c/titanic/data\n",
    "\n",
    "Features:\n",
    "- **PassengerId:** Id of every passenger.\n",
    "- **Survived:** This feature have value 0 and 1. 0 for not survived and 1 for survived.\n",
    "- **Pclass:** There are 3 classes of passengers. Class1, Class2 and Class3.\n",
    "- **Name:** Name of passenger.\n",
    "- **Sex:** Gender of passenger.\n",
    "- **Age:** Age of passenger.\n",
    "- **SibSp:** Indication that passenger have siblings and spouse.\n",
    "- **Parch:** Whether a passenger is alone or have family.\n",
    "- **Ticket:** Ticket no of passenger.\n",
    "- **Fare:** Indicating the fare.\n",
    "- **Cabin:** The cabin of passenger.\n",
    "- **Embarked:** The embarked category.\n",
    "- **Initial:** Initial name of passenger."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "\n",
    "titanic = pandas.read_csv('data/titanic.csv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "titanic[['Age', 'Fare']].boxplot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passenger gender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic['Sex'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic['Sex'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic['Sex'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot\n",
    "\n",
    "titanic['Sex'].value_counts().plot(kind='bar')\n",
    "\n",
    "pyplot.title('Number of Titanic passengers by gender')\n",
    "pyplot.ylabel('Number of passengers');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic['Sex'].replace({'male': 'M', 'female': 'F'}).value_counts().plot(kind='bar')\n",
    "\n",
    "pyplot.title('Number of Titanic passengers by gender')\n",
    "pyplot.ylabel('Number of passengers')\n",
    "pyplot.xticks(rotation=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <p><b>EXERCISE 1:</b> Checking where the passengers embarked</p>\n",
    "    <img alt=\"\" src=\"img/titanic_route.png\"/>\n",
    "    <p>Tasks:\n",
    "        <ul>\n",
    "            <li>Check the names of the ports in the dataset.</li>\n",
    "            <li>Replace the abbreviation of the ports by the full name.</li> \n",
    "            <li>Compute the number of passangers that embarked in each port.</li>\n",
    "            <li>Compute the proportion of passangers that embarked in each port.</li>\n",
    "            <li>Plot the proportion of passangers that embarked in each port.</li>\n",
    "        </ul>\n",
    "    </p>\n",
    "    <p>Hints:\n",
    "        <ul>\n",
    "            <li>The column <code>Embarked</code> has the port where the passenger embarked.</li>\n",
    "            <li>Unique values in a column can be obtained with <code>.unique()</code>.</li>\n",
    "            <li>The method <code>.replace()</code> receives a dictionary with lookups, and returns\n",
    "                the column with the values replaced.</li>\n",
    "            <li>To set a column with another (possibly calculated) column, you can use the method\n",
    "                <code>.assign()</code>.</li>\n",
    "            <li>The method <code>plot()</code> supports different <code>kind</code> of plots,\n",
    "                such as <code>bar</code>, <code>barh</code> and <code>pie</code>.</li>\n",
    "        </ul>\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/titanic_1.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passenger classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.pivot_table(values='PassengerId', index='Pclass', columns='Survived', aggfunc='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(titanic.assign(Survived=titanic['Survived'].replace({0: 'No', 1: 'Yes'}))\n",
    "        .pivot_table(values='PassengerId', index='Pclass', columns='Survived', aggfunc='count')\n",
    "        .loc[:, ['Yes', 'No']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <p><b>EXERCISE 2:</b> Checking survival by sex and class</p>\n",
    "    <img alt=\"\" src=\"img/titanic_classes.jpg\"/>\n",
    "    <p>Tasks:\n",
    "        <ul>\n",
    "            <li>Display a table with the passenger class as columns, and sex as rows.</li>\n",
    "            <li>Compute the proportion of passengers who survived in each class.</li> \n",
    "            <li>Format the proportions as percentages (e.g. <code>0.5</code> -> <code>50%</code>).</li>\n",
    "            <li>Rename the column <code>Pclass</code> to <code>Class</code> in the visualization.</li>\n",
    "        </ul>\n",
    "    </p>\n",
    "    <p>Hints:\n",
    "        <ul>\n",
    "            <li>The proportion of passangers who survived is <code>survived / total</code> and can be\n",
    "                computed as <code>titanic.Survived.sum() / titanic.Survived.count()</code>, which is equivalent to\n",
    "                <code>titanic.Survived.mean()</code>.</li>\n",
    "            <li>The method <code>.pivot_table()</code> accepts the parameter <code>aggfunc='mean'</code>.</li>\n",
    "            <li>The values of a table/DataFrame can be formatted as percentage calling\n",
    "                <code>.style.format('{:.2%}')</code>.</li>\n",
    "            <li>Columns can be renamed with the method <code>.rename()</code> using the parameter\n",
    "                <code>columns=</code> with a dictionary of the columns to rename.</code>\n",
    "        </ul>\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/titanic_2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passenger names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic['Name'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = 'Futrelle, Mrs. Jacques Heath (Lily May Peel)'\n",
    "name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splitted_names = name.split(',')\n",
    "splitted_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reversed_splitted_names = splitted_names[::-1]\n",
    "reversed_splitted_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_names = ' '.join(reversed_splitted_names)\n",
    "joined_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_names.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_names = (titanic['Name'].str.split(',')\n",
    "                             .str[::-1]\n",
    "                             .str.join(' ')\n",
    "                             .str.strip())\n",
    "full_names.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_names.str.startswith('Rev.').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_names[full_names.str.startswith('Rev.')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic[full_names.str.startswith('Rev.')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\">\n",
    "    <p><b>EXERCISE 3:</b> Who was the Titanic Captain?</p>\n",
    "    <img alt=\"\" src=\"img/titanic_captain.jpg\"/>\n",
    "    <p>Tasks:\n",
    "        <ul>\n",
    "            <li>Obtain a Series with the titles (e.g. Miss, Mr...) from the names of the passengers,\n",
    "                and how many of each exist in the data.</li>\n",
    "            <li>Identify which title corresponds to the captain.</li> \n",
    "            <li>Filter the original dataset to return only the row with the captain.</li>\n",
    "            <li>Visualize the captain name, age, and in which class he was travelling.</li>\n",
    "        </ul>\n",
    "    </p>\n",
    "    <p>Hints:\n",
    "        <ul>\n",
    "            <li>A DataFrame can be filtered by both rows and columns at the same time,\n",
    "                using the method <code>.loc[rows_filter, columns_filter]</code>.</li>\n",
    "            <li>A filter can be a label, a list of labels, a boolean array, or a slice.</li>\n",
    "        </ul>\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load solutions/titanic_3.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}