{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lab 03: Linear and logistic regressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this lab is to explore linear and logistic regression, implement them yourself and learn to use their respective scikit-learn implementation.\n",
    "\n",
    "Let us start by loading some of the usual librairies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Linear regression\n",
    "\n",
    "We will now implement a linear regression, first using the closed form solution, and second with our gradient descent.\n",
    "\n",
    "## 1.1 Linear regression data\n",
    "\n",
    "Our first data set regards the quality ratings of a white _vinho verde_. Each wine is described by a number of physico-chemical descriptors such as acidity, sulfur dioxide content, density or pH."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the regression task data\n",
    "wine_data = pd.read_csv('data/winequality-white.csv', sep=\";\")\n",
    "wine_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data into X and y data arrays\n",
    "X_regr = wine_data.drop(['quality'], axis=1).values\n",
    "y_regr = wine_data['quality'].values\n",
    "\n",
    "# Standardize the data\n",
    "from sklearn import preprocessing\n",
    "sc = preprocessing.StandardScaler()\n",
    "sc.fit(X_regr)\n",
    "X_regr = sc.transform(X_regr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Cross-validation\n",
    "\n",
    "Let us create a cross-validation utility function (similar to what we have done in Lab 3, but for regression)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up folds for cross_validation\n",
    "from sklearn import model_selection\n",
    "folds_regr = model_selection.KFold(n_splits=10, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validate_regr(design_matrix, labels, regressor, cv_folds):\n",
    "    \"\"\" Perform a cross-validation and returns the predictions.\n",
    "    \n",
    "    Parameters:\n",
    "    -----------\n",
    "    design_matrix: (n_samples, n_features) np.array\n",
    "        Design matrix for the experiment.\n",
    "    labels: (n_samples, ) np.array\n",
    "        Vector of labels.\n",
    "    regressor:  Regressor instance; must have the following methods:\n",
    "        - fit(X, y) to train the regressor on the data X, y\n",
    "        - predict(X) to apply the trained regressor to the data X and return estimates \n",
    "    cv_folds: sklearn cross-validation object\n",
    "        Cross-validation iterator.\n",
    "        \n",
    "    Returns:\n",
    "    -------\n",
    "    pred: (n_samples, ) np.array\n",
    "        Vectors of predictions (same order as labels).\n",
    "    \"\"\"\n",
    "    pred = np.zeros(labels.shape)\n",
    "    for tr, te in cv_folds:\n",
    "        regressor.fit(design_matrix[tr,:], labels[tr])\n",
    "        pred[te] = (regressor.predict(design_matrix[te,:]))\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Linear regression with scikit-learn\n",
    "\n",
    "__Question__ Cross-validate scikit-learn's [linear_model.LinearRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) on your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "# Initialize a LinearRegression model\n",
    "regr = # TODO\n",
    "\n",
    "# Cross-validate it\n",
    "pred = # TODO\n",
    "\n",
    "from sklearn import metrics\n",
    "print(\"Mean squared error: %.3f\" % metrics.mean_squared_error(y_regr, pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Logistic regression\n",
    "\n",
    "We will now implement a linear regression, first using the closed form solution, and second with our gradient descent.\n",
    "\n",
    "## 2.1 Logistic regression data\n",
    "\n",
    "Our second data set comes from the world of bioinformatics. In this data set, each observation is a tumor, and it is described by the expression of 3,000 genes. The expression of a gene is a measure of how much of that gene is present in the biological sample. Because this affects how much of the protein this gene codes for is produced, and because proteins dictacte what cells can do, gene expression gives us valuable information about the tumor. In particular, the expression of the same gene in the same individual is different in different tissues (although the DNA is the same): this is why blood cells look different from skin cells. In our data set, there are two types of tumors: breast tumors and ovary tumors. Let us see if gene expression can be used to separate them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the classification task data\n",
    "breast_data = pd.read_csv('data/small_Breast_Ovary.csv')\n",
    "\n",
    "# Drop the 'Tissue' column to create the design matrix\n",
    "X_clf = np.array(breast_data.drop(['Tissue', 'ID_REF'], axis=1).values)\n",
    "\n",
    "# Use the 'Tissue' column to create the labels (0=Breast, 1=Ovary)\n",
    "y_clf = np.array(breast_data['Tissue'].values)\n",
    "y_clf[np.where(y_clf == 'Breast')] = 0\n",
    "y_clf[np.where(y_clf == 'Ovary')] = 1\n",
    "y_clf = y_clf.astype(np.int)\n",
    "\n",
    "#sc = preprocessing.StandardScaler()\n",
    "#sc.fit(X_clf)\n",
    "#X_clf = sc.transform(X_clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question:__ How many samples do we have? How many belong to each class? How many features do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Cross-validation\n",
    "\n",
    "Let us create a cross-validation utility function (similar to what we have done in Lab 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up folds for cross_validation\n",
    "from sklearn import model_selection\n",
    "folds_clf = model_selection.StratifiedKFold(n_splits=10, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validate_clf(design_matrix, labels, classifier, cv_folds):\n",
    "    \"\"\" Perform a cross-validation and returns the predictions.\n",
    "    \n",
    "    Parameters:\n",
    "    -----------\n",
    "    design_matrix: (n_samples, n_features) np.array\n",
    "        Design matrix for the experiment.\n",
    "    labels: (n_samples, ) np.array\n",
    "        Vector of labels.\n",
    "    classifier:  sklearn classifier object\n",
    "        Classifier instance; must have the following methods:\n",
    "        - fit(X, y) to train the classifier on the data X, y\n",
    "        - predict_proba(X) to apply the trained classifier to the data X and return probability estimates \n",
    "    cv_folds: sklearn cross-validation object\n",
    "        Cross-validation iterator.\n",
    "        \n",
    "    Return:\n",
    "    -------\n",
    "    pred: (n_samples, ) np.array\n",
    "        Vectors of predictions (same order as labels).\n",
    "    \"\"\"\n",
    "    pred = np.zeros(labels.shape)\n",
    "    for tr, te in cv_folds:\n",
    "        classifier.fit(design_matrix[tr,:], labels[tr])\n",
    "        pred[te] = classifier.predict_proba(design_matrix[te,:])[:,1]\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Logistic regression with scikit-learn\n",
    "\n",
    "__Question__ Cross-validate scikit-learn's [linear_model.LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) on your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "# Initialize a LogisticRegression model. \n",
    "# Use C=1e7 to ensure there is no regularization (we'll talk about regularization next time!)\n",
    "clf = # TODO\n",
    "\n",
    "# Cross-validate it\n",
    "ypred_logreg = # TODO\n",
    "\n",
    "print(\"Accuracy: %.3f\" % metrics.accuracy_score(ypred_logreg > 0.5, 1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Question : ** Plot the ROC curve. Use plt.semilogx to use a logarithmic scale on the x-axis. This \"spreads out\" the curve a little, making it easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr_logreg, tpr_logreg, thresholds = # TODO\n",
    "auc_logreg = # TODO\n",
    "\n",
    "plt.semilogx(fpr_logreg, tpr_logreg, '-', color='orange', \n",
    "             label='AUC = %0.3f' % auc_logreg)\n",
    "\n",
    "plt.xlabel('False Positive Rate', fontsize=16)\n",
    "plt.ylabel('True Positive Rate', fontsize=16)\n",
    "plt.title('ROC curve: Logistic regression', fontsize=16)\n",
    "plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data scaling\n",
    "See [preprocessing.StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** Scale the data, and compute the cross-validated predictions of the logistic regression on the scaled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "# Scale the data with preprocessing.StandardScaler\n",
    "# Initialize a scaler\n",
    "scaler = # TODO\n",
    "# Scale your design matrix\n",
    "X_clf_scaled = # TODO\n",
    "\n",
    "# Initialize a LogisticRegression model. \n",
    "# Use C=1e7 to ensure there is no regularization (we'll talk about regularization next time!)\n",
    "clf = # TODO\n",
    "\n",
    "# Cross-validate it for the scaled data\n",
    "ypred_logreg_scaled = # TODO\n",
    "\n",
    "print(\"Accuracy: %.3f\" % metrics.accuracy_score(ypred_logreg_scaled > 0.5, 1, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** Plot the two ROC curves (one for the logistic regression on the original data, one for the logistic regression on the scaled data) on the same plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr_logreg_scaled, tpr_logreg_scaled, thresholds = # TODO\n",
    "auc_logreg_scaled = # TODO\n",
    "\n",
    "plt.semilogx(fpr_logreg_scaled, tpr_logreg_scaled, '-', color='blue', \n",
    "             label='scaled data; AUC = %0.3f' % auc_logreg_scaled)\n",
    "plt.semilogx(fpr_logreg, tpr_logreg, '-', color='orange', \n",
    "             label='original data; AUC = %0.3f' % auc_logreg)\n",
    "\n",
    "plt.xlabel('False Positive Rate', fontsize=16)\n",
    "plt.ylabel('True Positive Rate', fontsize=16)\n",
    "plt.title('ROC curve: Logistic regression', fontsize=16)\n",
    "plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a cross-validation setting, we ignore the samples from the test fold when training the classifier. This also means that scaling should be done on the training data only. \n",
    "\n",
    "In scikit-learn, we can use a scaler to make centering and scaling happen independently on each feature by computing the relevant statistics on the samples *in the training set*. \n",
    "The mean and standard deviation will be stored to be used on the test data.\n",
    "\n",
    "**Question** Rewrite the cross_validate method to include a scaling step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validate_clf_with_scaling(design_matrix, labels, classifier, cv_folds):\n",
    "    \"\"\" Perform a cross-validation and returns the predictions.\n",
    "    \n",
    "    Parameters:\n",
    "    -----------\n",
    "    design_matrix: (n_samples, n_features) np.array\n",
    "        Design matrix for the experiment.\n",
    "    labels: (n_samples, ) np.array\n",
    "        Vector of labels.\n",
    "    classifier:  sklearn classifier object\n",
    "        Classifier instance; must have the following methods:\n",
    "        - fit(X, y) to train the classifier on the data X, y\n",
    "        - predict_proba(X) to apply the trained classifier to the data X and return probability estimates \n",
    "    cv_folds: sklearn cross-validation object\n",
    "        Cross-validation iterator.\n",
    "        \n",
    "    Return:\n",
    "    -------\n",
    "    pred: (n_samples, ) np.array\n",
    "        Vectors of predictions (same order as labels).\n",
    "    \"\"\"\n",
    "    pred = np.zeros(labels.shape)\n",
    "    for tr, te in cv_folds:\n",
    "        # TODO\n",
    "    return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** Now use the cross_validate_with_scaling method to cross-validate the logistic regression on our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = linear_model.LogisticRegression(C=1e6) \n",
    "ypred_logreg_scaled_ = cross_validate_clf_with_scaling(X_clf, y_clf, clf, folds_clf)\n",
    "print(metrics.accuracy_score(y_clf,np.where(ypred_logreg_scaled_ > 0.5, 1, 0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question** Again, compare the AUROC and ROC curves with those obtained previously. What do you conclude?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr_logreg_scaled_, tpr_logreg_scaled_, thresholds = # TODO\n",
    "auc_logreg_scaled_ = # TODO\n",
    "\n",
    "plt.semilogx(fpr_logreg_scaled, tpr_logreg_scaled, '-', \n",
    "             color='blue', label='scaled data overfit; AUC = %0.3f' % auc_logreg_scaled)\n",
    "plt.semilogx(fpr_logreg, tpr_logreg, '-', color='orange', \n",
    "             label='original data; AUC = %0.3f' % auc_logreg)\n",
    "plt.semilogx(fpr_logreg_scaled_, tpr_logreg_scaled_, '-', color='black', \n",
    "             label='scaled data no overfit; AUC = %0.3f' % auc_logreg_scaled_)\n",
    "\n",
    "\n",
    "plt.xlabel('False Positive Rate', fontsize=16)\n",
    "plt.ylabel('True Positive Rate', fontsize=16)\n",
    "plt.title('ROC curve: Logistic regression', fontsize=16)\n",
    "plt.legend(loc=\"lower right\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}