{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Home Page](../Start_Here.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;\n",
    "[Next Notebook](CNN's.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNN Primer and Keras 101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this notebook, participants will be introduced to CNN, implement it using Keras. For an absolute beginner this notebook would serve as a good starting point.\n",
    "\n",
    "**Contents of the this notebook:**\n",
    "\n",
    "- [How a Deep Learning project is planned ?](#Machine-Learning-Pipeline)\n",
    "- [Wrapping things up with an example ( Classification )](#Wrapping-Things-up-with-an-Example)\n",
    "     - [Fully Connected Networks](#Image-Classification-on-types-of-Clothes)\n",
    "\n",
    "\n",
    "**By the end of this notebook participant will:**\n",
    "\n",
    "- Understand the Machine Learning Pipeline\n",
    "- Write a Deep Learning Classifier and train it.\n",
    "\n",
    "**We will be building a _Multi-class Classifier_ to classify images of clothing to their respective classes**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Pipeline\n",
    "\n",
    "During the bootcamp we will be making use of the following buckets to help us understand how a Machine Learning project should be planned and executed: \n",
    "\n",
    "1. **Data**: To start any ML project we need data which is pre-processed and can be fed into the network.\n",
    "2. **Task**: There are many tasks present in ML, we need to make sure we understand and define the problem statement accurately.\n",
    "3. **Model**: We need to build our model, which is neither too deep and taking a lot of computational power or too small that it could not learn the important features.\n",
    "4. **Loss**: Out of the many _loss functions_ present, we need to carefully choose a _loss function_ which is suitable for the task we are about to carry out.\n",
    "5. **Learning**: As we mentioned in our last notebook, there are a variety of _optimisers_ each with their advantages and disadvantages. So here we choose an _optimiser_ which is suitable for our task and train our model using the set hyperparameters.\n",
    "6. **Evaluation**: This is a crucial step in the process to determine if our model has learnt the features properly by analysing how it performs when unseen data is given to it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here we will be building a _Multi-class Classifier_ to classify images of clothing to their respective classes.**\n",
    "\n",
    "We will follow the above discussed pipeline to complete the example.\n",
    "\n",
    "## Image Classification on types of clothes  \n",
    "\n",
    "####  Step -1 : Data \n",
    "\n",
    "We will be using the **F-MNIST ( Fashion MNIST )** dataset, which is a very popular dataset. This dataset contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels).\n",
    "\n",
    "<img src=\"images/fashion-mnist.png\" alt=\"Fashion MNIST sprite\"  width=\"600\">\n",
    "\n",
    "*Source: https://www.tensorflow.org/tutorials/keras/classification*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Necessary Libraries\n",
    "\n",
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "# TensorFlow and tf.keras\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "# Helper libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Let's Import the Dataset\n",
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "\n",
    "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the dataset returns four NumPy arrays:\n",
    "\n",
    "* The `train_images` and `train_labels` arrays are the *training set*—the data the model uses to learn.\n",
    "* The model is tested against the *test set*, the `test_images`, and `test_labels` arrays.\n",
    "\n",
    "The images are 28x28 NumPy arrays, with pixel values ranging from 0 to 255. The *labels* are an array of integers, ranging from 0 to 9. These correspond to the *class* of clothing the image represents:\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <th>Label</th>\n",
    "    <th>Class</th>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>0</td>\n",
    "    <td>T-shirt/top</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>1</td>\n",
    "    <td>Trouser</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>2</td>\n",
    "    <td>Pullover</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>3</td>\n",
    "    <td>Dress</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>4</td>\n",
    "    <td>Coat</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>5</td>\n",
    "    <td>Sandal</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>6</td>\n",
    "    <td>Shirt</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>7</td>\n",
    "    <td>Sneaker</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>8</td>\n",
    "    <td>Bag</td>\n",
    "  </tr>\n",
    "    <tr>\n",
    "    <td>9</td>\n",
    "    <td>Ankle boot</td>\n",
    "  </tr>\n",
    "</table>\n",
    "\n",
    "Each image is mapped to a single label. Since the *class names* are not included with the dataset, let us store them in an array so that we can use them later when plotting the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
    "               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Print Array Size of Training Set \n",
    "print(\"Size of Training Images :\"+str(train_images.shape))\n",
    "#Print Array Size of Label\n",
    "print(\"Size of Training Labels :\"+str(train_labels.shape))\n",
    "\n",
    "#Print Array Size of Test Set \n",
    "print(\"Size of Test Images :\"+str(test_images.shape))\n",
    "#Print Array Size of Label\n",
    "print(\"Size of Test Labels :\"+str(test_labels.shape))\n",
    "\n",
    "#Let's See how our Outputs Look like \n",
    "print(\"Training Set Labels :\"+str(train_labels))\n",
    "#Data in the Test Set\n",
    "print(\"Test Set Labels :\"+str(test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Pre-processing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.imshow(train_images[0])\n",
    "plt.colorbar()\n",
    "plt.grid(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image pixel values range from 0 to 255. Let us now normalise the data range from 0 - 255 to 0 - 1 in both the *Train* and *Test* set. This Normalisation of pixels helps us by optimizing the process where the gradients are computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_images = train_images / 255.0\n",
    "test_images = test_images / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's Print to Veryify if the Data is of the correct format.\n",
    "plt.figure(figsize=(10,10))\n",
    "for i in range(25):\n",
    "    plt.subplot(5,5,i+1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(train_images[i], cmap=plt.cm.binary)\n",
    "    plt.xlabel(class_names[train_labels[i]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining our Model\n",
    "\n",
    "Our Model has three layers :\n",
    "\n",
    "- 784 Input features ( 28 * 28 ) \n",
    "- 128 nodes in hidden layer (Feel free to experiment with the value)\n",
    "- 10 output nodes to denote the Class\n",
    "\n",
    "Implementing the same in Keras ( Machine Learning framework built on top of Tensorflow, Theano, etc..) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import backend as K\n",
    "K.clear_session()\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Flatten(input_shape=(28, 28)),\n",
    "    keras.layers.Dense(128, activation='relu'),\n",
    "    keras.layers.Dense(10, activation='softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first layer in this network, `tf.keras.layers.Flatten`, transforms the format of the images from a two-dimensional array (of 28 by 28 pixels) to a one-dimensional array (of 28 * 28 = 784 pixels). Think of this layer as unstacking rows of pixels in the image and lining them up. This layer has no parameters to learn; it only reformats the data.\n",
    "\n",
    "After the pixels are flattened, the network consists of a sequence of two `tf.keras.layers.Dense` layers. These are densely connected, or fully connected, neural layers. The first `Dense` layer has 128 nodes (or neurons). The second (and last) layer is a 10-node *softmax* layer that returns an array of 10 probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the 10 classes.\n",
    "\n",
    "### Compile the model\n",
    "\n",
    "Before the model is ready for training, it needs a few more settings. These are added during the model's *compile* step:\n",
    "\n",
    "* *Loss function* —This measures how accurate the model is during training. You want to minimize this function to \"steer\" the model in the right direction.\n",
    "* *Optimizer* —This is how the model is updated based on the data it sees and its loss function.\n",
    "* *Metrics* —Used to monitor the training and testing steps. The following example uses *accuracy*, the fraction of the images that are correctly classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "Training the neural network model requires the following steps:\n",
    "\n",
    "1. Feed the training data to the model. In this example, the training data is in the `train_images` and `train_labels` arrays.\n",
    "2. The model learns to associate images and labels.\n",
    "3. You ask the model to make predictions about a test set—in this example, the `test_images` array. Verify that the predictions match the labels from the `test_labels` array.\n",
    "\n",
    "To start training,  call the `model.fit` method—so called because it \"fits\" the model to the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(train_images, train_labels ,epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate accuracy\n",
    "\n",
    "Next, compare how the model performs on the test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Evaluating the Model using the Test Set\n",
    "\n",
    "test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)\n",
    "\n",
    "print('\\nTest accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an Accuracy of 87% in the Test dataset which is less than the 89% we got during the Training phase, This problem in ML is called as Overfitting, and we have discussed the same in the previous notebook. \n",
    "\n",
    "## Exercise\n",
    "\n",
    "Try adding more dense layers to the network above and observe change in accuracy.\n",
    "\n",
    "## Important:\n",
    "<mark>Shutdown the kernel before clicking on “Next Notebook” to free up the GPU memory.</mark>\n",
    "\n",
    "\n",
    "## Licensing\n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Home Page](../Start_Here.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;\n",
    "[Next Notebook](CNN's.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
