{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this notebook, you will be introduced to Machine Learning(ML), and it's terminologies. For an absolute beginner this notebook would serve as an excellent starting point.\n",
    "\n",
    "**Contents of the This Notebook:**\n",
    "\n",
    "- [What is Machine Learning?](#Machine-Learning)\n",
    "- [Neural Networks](#Neural-Networks)\n",
    "- [Model](#Model) \n",
    "- [Tasks in Machine Learning](#Tasks-in-Machine-Learning)\n",
    "- [Forward Propogation](#Forward-Propagation)\n",
    "- [Loss Function](#Loss-Functions)\n",
    "- [Backpropagation](#Backpropagation)\n",
    "- [Optimizers](#Optimizers)\n",
    "- [Hyper Parameters](#Hyper-Parameters)\n",
    "- [Training & Evaluation](#Training-&-Evaluation)\n",
    "\n",
    "\n",
    "**By the end of this Notebook you will:**\n",
    "\n",
    "- Understand Machine Learning Terminologies\n",
    "- Understand how Neural Networks work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning\n",
    "\n",
    "Machine Learning is a subset of Artificial Intelligence (AI) and a scientific field of study which uses _Algorithms_ and _Statistics_ to perform specific _Tasks_ without _Explicit Instructions_ of how that has to be performed. This is done by statistical pattern recognition and inferencing relevant information from data.\n",
    "\n",
    "As of Today, Machine Learning is used widely in a variety of fields from [Self-Driving Cars](https://blogs.nvidia.com/blog/2019/08/21/drive-labs-autonomous-vehicle-ride/), [Identifying Diseases in the Healthcare Industry](https://news.developer.nvidia.com/ai-automatically-detects-lung-abnormalities/) to [Generating Photorealistic Image from Doodles](https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealistic-landscapes-nvidia-research/?ncid=so-you-n1-78256). \n",
    "\n",
    "Machine Learning is broadly classified Into three types:\n",
    "- *Supervised Learning*\n",
    "    - Given _Inputs_ and corresponding _Outputs_, the algorithms learns to _Map_ a _Function_ between _Inputs_ and _Outputs_ over an iterative optimization process, also called as _Training_. Example: You provide pictures of Cats and Dogs with labels of them and the algorithms can learn the relation between the input and the output. Algorithms that fall in this category are Support Vector Machine, Decision Trees and many other. \n",
    "- *Unsupervised Learning*\n",
    "    - We do not give any Targets (Labels) to the System, and it comes up with its way of understanding the data. Example : You give a bunch of Images of cats and dogs and you want to divide it into two piles , the algorithms goes through the images and learns to divide the images into these piles on it own. Algorithms that fall in this catgory are k-means, clustering and many other \n",
    "- *Reinforcement Learning*\n",
    "    - Reinforcement learning differs from other algorithms in a way that the training data has the answer key with it, so the model is trained with the correct answer itself whereas, in reinforcement learning, there is no answer, but the reinforcement agent decides what to do to perform the given task. In the absence of training dataset, it is bound to learn from its experience. Example include recent real-life achievements such as : AlphaGo is the first computer program to defeat a professional human Go player. \n",
    "    \n",
    "![alt_text](images/sup-unsup.png)\n",
    "    \n",
    "    \n",
    "**There are many different algorithms for each domain. In this notebook we will be limiting ourselves to Supervised Learning and also we will be limiting our focus to one specific algorithm called the Artifical Neural Networks using which we will introduce the concepts of Deep Learning.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "Neural Networks are a set of algorithms modelled after human brain to recognize patterns.\n",
    "\n",
    "The human brain consists of neurons or nerve cells which transmit and process the information received from our senses. Many such nerve cells are arranged together in our brain to form a network of nerves. These nerves pass electrical impulses, i.e. the excitation from one neuron to the other.\n",
    "\n",
    "*The same concept of the network of neurons is used in machine learning algorithms. In this case, the neurons are created artificially on a computer. Connecting many such artificial neurons creates an artificial neural network.*\n",
    "\n",
    "\n",
    "<table><tr>\n",
    "<td> <img src=\"images/neuron.jpg\" alt=\"Drawing\" style=\"width: 540px;\"/></td>\n",
    "<td> <img src=\"images/ann.png\" alt=\"Drawing\" style=\"width: 500px;\"/> </td>\n",
    "</tr></table>\n",
    "\n",
    "\n",
    "**How it works:**\n",
    "\n",
    "$x1,x2$ are inputs to the Neuron while $y$ is the output of the Neuron.This $x1,x2$ are then multiplied with $w1,w2$ respectively. \n",
    "\n",
    "$w1,w2$ can be referred here as the Weight of the input Features $x1,x2$ , It is this $w1,w2$ that is learned by the Algorithm over time.\n",
    "\n",
    "This Weight Matrix along with the Input Matrix will be applied on by an Activation Function ( $f(x)$ ) \n",
    "\n",
    "$ y= f ( x1 * w1 + x2 * w2 ) $\n",
    "\n",
    "This output will be fed into further Neuron.\n",
    "\n",
    "\n",
    "### What is an Activation Function?\n",
    "\n",
    "When multiple layers are present, and an Input $x1,x2,x3\\dots,xn$ is fed into the network, it is not necessary for the output to be a Linear Function of $x1,x2,x3\\dots xn$ such as\n",
    "\n",
    "$ y = a1*x1 + a2*x2 + \\dots + an*xn $ \n",
    "\n",
    "\n",
    "Why is this a Problem? Because lots of functions in the real-world are not necessarily linear. Think of a Classifier that has to classify if a specific latitude and longitude is part of the city or outside a city. In that case, the function can look something like this.\n",
    "\n",
    "<img src=\"images/location.PNG\" alt=\"Drawing\" style=\"width: 300px;\"/>\n",
    "\n",
    "To classify such a function that is non-linear in nature, we need to introduce some non-linearity into the network to learn this complex function. This is where the Activation Function helps us by Activating the Neurons which has contribution to the output.\n",
    "\n",
    "Some Popular Activation Functions are :\n",
    "\n",
    "- *Sigmoid Function*\n",
    "- *tanh*\n",
    "- *ReLU*\n",
    "- *Leaky ReLU* \n",
    "- *ELU* \n",
    "<img src=\"images/activation_fns.png\" alt=\"Drawing\" style=\"width: 400px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "A Neural Network Model/Architecture is made of multiple layers which are built by combining the artifical neurons. The three fundamental types of layers are: \n",
    "\n",
    "- *Input Layers:*\n",
    "    - Initial Layer where the Data/Input is fed into the Neural Network\n",
    "- *Hidden Layers:*\n",
    "    - Intermediate layers between input and output layer where the computation takes place.\n",
    "- *Output Layers:*\n",
    "    - Final Layers of the Network which produces the output for the given inputs\n",
    "    \n",
    "Here is an example of a Network with the Following Parameters : \n",
    "\n",
    "- **Input Layer** : 3 Input Nodes\n",
    "- **Hidden Layer 1** : 4 Nodes\n",
    "- **Hidden Layer 2** : 4 Nodes\n",
    "- **Output Layer** : 1 Output Node\n",
    "\n",
    "<img src=\"images/model.png\" alt=\"Drawing\" style=\"width: 400px;\"/>\n",
    "\n",
    "Neural Network Models are built based on the dimensions of input data and the complexity of the task it is supposed to solve. Let us see some of the common tasks where Neural Networks are widely used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tasks in Machine Learning\n",
    "\n",
    "There are different types of _Tasks_ in Machine Learning. Some of the popular ones are as follows:\n",
    "\n",
    "- *Regression*\n",
    "    - Predicting a dependent value Y given an independent variable X. One such example is predicting housing prices given the location of the property.\n",
    "- *Classification*\n",
    "    - Predicting a class to which an object belongs to, such as predicting if the image given is of a Dog /Cat, or what letter was spelt in Voice Recognition applications.\n",
    "- *Clustering*\n",
    "    - Clustering is the task of dividing the population or data points into several groups such that data points in the same groups are more similar to other data points in the same group and dissimilar to the data points in other groups.\n",
    "- *Dimension reduction*\n",
    "    - Dimensionality reduction is the process of reducing the number of random variables under consideration by obtaining a set of principal variables.\n",
    "\n",
    "Once the _Task_ (Problem Statement) is clear to us, and a Model is defined, we train the model using the _Input_ and _Output_ data we have. And to understand how training works, let us get ourselves introduced on how data flows in a model from the _Input Layer_ to the _Output Layer_ and vice-versa."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward Propagation\n",
    "\n",
    "Forward propagation, as the name suggests, is the process where the input data is propagated in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function and passes to the successive layer until the data reaches the output layer.\n",
    "\n",
    "Let us illustrate it by an example. Let us define our Activation Functions to be the following :\n",
    "\n",
    "$$ f(x)=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & x<0 \\\\\n",
    "      1 & x>=0 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "#### OR Gate Forward Propagation\n",
    "\n",
    "Let us define the OR Gate as a Neural Network.\n",
    "\n",
    "<table><tr>\n",
    "<td> <img src=\"images/or_left.gif\" alt=\"Drawing\" style=\"width: 400px;\"/></td>\n",
    "<td> <img src=\"images/or_right.png\" alt=\"Drawing\" style=\"width: 400px;\"/> </td>\n",
    "</tr></table>\n",
    "\n",
    "Let us now take our inputs to be $x1$ and $x2$ and multiply it with their respective weights. \n",
    "\n",
    "Note: $b$ stands for bias ( Bias is a constant which helps the model in a way that it can fit best for the given data) \n",
    "\n",
    "*Feedforwarding the Input Layer to the Second layer :*\n",
    "\n",
    "$a1(2) = x1 * 1 + x2 *1 -1  = x1+x2-1$   \n",
    "\n",
    "Applying our Activation Function : \n",
    "\n",
    "$$ OR = f(a1(2))=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & x1+x2-1<0 \\\\\n",
    "      1 & x1+x2-1>=0 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "And can be re-written as\n",
    "\n",
    "$$ OR = f(a1(2))=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & x1+x2<1 \\\\\n",
    "      1 & x1+x2>=1 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "$x1+x2 $ can only be less than 1 if both $x1,x2$ are 0. Hence we obtain the following table\n",
    "\n",
    "| X1 | X2 |  X1 OR X2 | OR ( From Neural Network ) |\n",
    "|----|----|-----------|----------------------------|\n",
    "|   0|   0|          0|                           0|\n",
    "|   0|   1|          1|                           1|\n",
    "|   1|   0|          1|                           1|\n",
    "|   1|   1|          1|                           1|\n",
    "\n",
    "#### XNOR Gate Forward Propagation\n",
    "\n",
    "Let us define the XNOR Gate as a Neural Network.\n",
    "\n",
    "<table><tr>\n",
    "<td> <img src=\"images/xnor_left.gif\" alt=\"Drawing\" style=\"width: 400px;\"/></td>\n",
    "<td> <img src=\"images/xnor_right.png\" alt=\"Drawing\" style=\"width: 400px;\"/> </td>\n",
    "</tr></table>\n",
    "\n",
    "Inputs $x1$ and $x2$ are then multiplied with their respective weights. $b$ stands for bias ( Bias is a constant which helps the model in a way that it can fit best for the given data) \n",
    "\n",
    "*Feedforwarding the input layer to the second layer :*\n",
    "\n",
    "$a1(2) = x1 * 1 + x2 *1 -1.5  = x1+x2-1.5$ \n",
    "\n",
    "$a2(2) = x1 * -1 + x2 *-1 + 0.5  = -x1-x2+0.5$ \n",
    "\n",
    "Applying our _Activation Function_ : \n",
    "\n",
    "$$ AND = f(a1(2))=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & x1+x2-1.5<0 \\\\\n",
    "      1 & x1+x2-1.5>=0 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "$$ NOR = f(a2(2))=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & -x1-x2+0.5<0 \\\\\n",
    "      1 & -x1-x2+0.5>=0 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "*Feedforwarding the second layer to the third layer :*\n",
    "\n",
    "$a1(3) = f(a1(2)) * 1 + f(a2(2)) *1 -1  = f(a1(2))+f(a2(2))-1$ \n",
    "\n",
    "\n",
    "$$ XNOR = f(a1(3))=   \\left\\{\n",
    "\\begin{array}{ll}\n",
    "      0 & f(a1(2))+f(a2(2))-1 < 0 \\\\\n",
    "      1 & f(a1(2))+f(a2(2))-1 >= 0 \\\\\n",
    "\\end{array} \n",
    "\\right.  $$\n",
    "\n",
    "Let's now compute the values : \n",
    "\n",
    "| X1 | X2 |  $$f(a1(2))$$| $$f(a2(2))$$ | $$f(a1(3))$$ |\n",
    "|----|----|-----------|-----------|-------|\n",
    "|   0|   0|          0|          1|      1|\n",
    "|   0|   1|          0|          0|0|\n",
    "|   1|   0|          0|          0|0|\n",
    "|   1|   1|          1|          0|1|\n",
    "\n",
    "\n",
    "So ,we obtain the following table :\n",
    "\n",
    "| X1 | X2 |  X1 XNOR X2 | XNOR ( From Neural Network ) |\n",
    "|----|----|-----------|----------------------------|\n",
    "|   0|   0|          1|                           1|\n",
    "|   0|   1|          0|                           0|\n",
    "|   1|   0|          0|                           0|\n",
    "|   1|   1|          1|                           1|\n",
    "\n",
    "\n",
    "\n",
    "What we saw in this section was a sample demonstration of how _Forward Propagation_ works with simple gates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss Functions\n",
    "\n",
    "In the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as the objective function.\n",
    "\n",
    "We may seek to maximize or minimize the objective function, meaning that we are searching for a candidate solution that has the highest or lowest score, respectively. Typically, with neural networks, we seek to minimize the error. As such, the objective function is often referred to as a cost function or a loss function, and the value calculated by the loss function is referred to as simply “loss.”\n",
    "\n",
    "The function we want to minimize or maximize is called the objective function or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function. The cost or loss function has an essential job to distill all aspects of the model down into a single number in such a way that improvements in that number are a sign of a better model.\n",
    "\n",
    "The cost function reduces all the various good and bad aspects of a possibly complex system down to a single number, a scalar value, which allows candidate solutions to be ranked and compared.\n",
    "\n",
    "Here are some of the Popular Loss Functions used for various different tasks in Machine Learning : \n",
    "- *Regression Loss Functions :*\n",
    "  - Mean Squared Error Loss\n",
    "  - Mean Squared Logarithmic Error Loss\n",
    "  - Mean Absolute Error Loss\n",
    "- *Binary Classification Loss Functions :*\n",
    "  - Binary Cross-Entropy\n",
    "  - Hinge Loss\n",
    "  - Squared Hinge Loss\n",
    "- *Multi-Class Classification Loss Functions :*\n",
    "  - Multi-Class Cross-Entropy Loss\n",
    "  - Sparse Multiclass Cross-Entropy Loss\n",
    "  - Kullback Leibler Divergence Loss\n",
    "  \n",
    "\n",
    "**(Optional Example )**  \n",
    "\n",
    "Let us Construct the *Mean Square Error* Loss Function , it is defined as follows : \n",
    "\n",
    "$$MSE = \\frac{\\sum_{i=1}^{n}(y_i - y_i^p)^2}{n}$$\n",
    "\n",
    "Where $y_i$ is the predicted value by the Network and $y_i^p $ is the Ground Truth value.\n",
    "\n",
    "Let us compare two models using the Loss Function.\n",
    "\n",
    "For Model-1, The predicted values is mentioned as $y_i^1 $, and respectively for Model-2, we can write it as $y_i^2$\n",
    "\n",
    "|$y_i^p$|Model -1 $y_i^1$| Squared Error $(y_i^1 - y_i^p)^2$| Model -2 $y_i^2$| Squared Error $(y_i^2 - y_i^p)^2$|\n",
    "|-------|-------|----------------------------------|-------|----------------------------------|\n",
    "|      1|    1.3|                              0.09|    1.1|                              0.01| \n",
    "|      2|    1.7|                              0.09|    2.1|                              0.01|\n",
    "|      3|    3.1|                              0.01|    3.2|                              0.04|                      \n",
    "|      4|    4.5|                              0.25|    3.8|                              0.04|\n",
    "\n",
    "Here, The $MSE$ turns out to be \n",
    "\n",
    "|Model-1|Model-2|\n",
    "|---|---|\n",
    "|$0.11$|$0.025$|\n",
    "\n",
    "Now, as we see the outputs from the models and their corresponding Loss values, it is intuitive to understand the loss function is a useful metric of understanding how the model improves over time during training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation\n",
    "\n",
    "If you notice, the OR Gate Model has the parameters $ -1, 1, 1 $ in the first layer, but these weights were hand-crafted to get the desired output ( in our case, the OR Gate). Typically, when a Model is defined, it is initialized with random weights. We consider a Model to have been trained when the weights are appropriately adjusted so that during the Forward Propagation the predicted outputs match with what we expect it to be. But how do the weights correct themselves if they were initialized randomly?\n",
    "\n",
    "This is where Backward Propagation of Error or \"Backpropagation\" as it is called helps us in propagating the gradients of the Error Functions from the Output Layer being calculated first and propagated back to the previous layers.\n",
    "\n",
    "Partial computations of the gradient from one layer are reused in the computation of the gradient for the previous layer. This backward flow of the error information allows for efficient computation of the gradient at each layer which is utilized by the optimiser to learn how much the weights have to be tweaked. You'll learn about Optimisers in the upcoming sections.\n",
    "\n",
    "\n",
    "<table><tr>\n",
    "<td> <img src=\"images/bp_left.gif\" alt=\"Drawing\" style=\"width: 540px;\"/></td>\n",
    "<td> <img src=\"images/bp_right.gif\" alt=\"Drawing\" style=\"width: 500px;\"/> </td>\n",
    "</tr></table>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizers\n",
    "\n",
    "During the training process, we tweak and change the parameters (weights) of our model to try and minimize that loss function, to make our predictions as correct as possible. But how exactly do you do that? How do you change the parameters of your model, by how much, and when?\n",
    "\n",
    "This is where optimizers come in. They tie together the loss function and model parameters by updating the model in response to the output of the loss function. In simpler terms, optimizers shape and mould your model into its most accurate possible form by futzing with the weights. The loss function is the guide to the terrain, telling the optimizer when it’s moving in the right or wrong direction.\n",
    "\n",
    "There are many types of Optimizers each with their respective advantages, drawbacks and application, where they find a great fit.\n",
    "\n",
    "Some of the Optimizers are :\n",
    "\n",
    "- Gradient Descent\n",
    "- Stochastic Gradient Descent\n",
    "- Adagrad\n",
    "- RMSprop\n",
    "- Adam\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper Parameters\n",
    "\n",
    "A model hyperparameter is a configuration that is external to the model and whose value cannot be estimated from data.\n",
    "\n",
    "They are often used in processes to help estimate model parameters and is specified by the practitioner.\n",
    "They can often be set using heuristics.\n",
    "They are often tuned for a given predictive modelling problem.\n",
    "We cannot know the best value for a model hyperparameter on a given problem. We may use rules of thumb, copy values used on other problems, or search for the best value by trial and error.\n",
    "\n",
    "When a machine learning algorithm is tuned for a specific problem, such as when you are using a grid search or a random search, then you are tuning the hyperparameters of the model or it's order to discover the parameters of the model that result in the most skillful predictions.\n",
    "\n",
    "The number and diversity of hyperparameters in machine learning algorithms are particular to each model. However, there some classic hyperparameters that we should always keep our eyes on and that should help you think about this aspect of machine learning solutions:\n",
    "\n",
    "- **Learning Rate**: The learning rate quantifies the learning progress of a model in a way that can be used to optimize its capacity. The learning rate also determine how we control the change to the weights, a lower value might take more time to converge while a large value might change the weights too much that we can miss a local minima.\n",
    "- **Epochs**: An epoch is defined as the number of times an algorithm visits the data set\n",
    "- **Batch Size**: The batch size is a hyperparameter of gradient descent that controls the number of training samples to work through before the model’s internal parameters are updated.\n",
    "- **The number of Hidden Units**: A classic hyperparameter in deep learning algorithms, the number of hidden units is vital to regulate the representational capacity of a model.\n",
    "- **Convolution Kernel Width**: In convolutional Neural Networks(CNNs), the Kernel Width influences the number of parameters in a model which, in turns, influences its capacity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training & Evaluation\n",
    "\n",
    "Training is an tterative optimization process when the _Objective Function_ ( Loss Function ) is minimized using the _Optimizer_ and the set hyper parameters.\n",
    "\n",
    "In layman terms , we can consider the Training process to be the following : \n",
    "\n",
    "```python3\n",
    "# Training Process  \n",
    "while ( Loss > Threshold ) :\n",
    "    Forward_Propogate()\n",
    "    Loss = Calculate_Loss()\n",
    "    Backpropogate_Loss_Gradients()\n",
    "    Optimize_the_weights()\n",
    "```\n",
    "\n",
    "<img src=\"images/training.gif\" alt=\"Drawing\" style=\"width: 700px;\"/> \n",
    "\n",
    "### Evalutaion of Model \n",
    "\n",
    "Model Evaluation is an integral part of the model development process. It helps to find the best model that represents our data and how well the chosen model will work in the future. \n",
    "\n",
    "In Machine Learning we want our _Model_ to have a robust fit over the data and not _Overfit_ or _Underfit_ the data.\n",
    "\n",
    "- **Underfitting** : When the _Model_ does not learn enough feature and is very simple to have properly understood the trend of the data.\n",
    "- **Overfitting** : When the _Model_ has learnt too much from the training data and has failed the _Generalise_ the data.\n",
    "\n",
    "![alt_text](images/evaluation.jpg)\n",
    "\n",
    "To Counter This , we split the data into three segments :\n",
    "\n",
    "- Training Dataset: The sample of data used to fit the model.\n",
    "- Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.\n",
    "- Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.\n",
    "\n",
    "The Test Accuracy of the Model is finally considered as the accuracy of the Model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References :\n",
    "\n",
    "This Notebook has contents inspired from the following references : \n",
    "\n",
    "[Artificial Neural Networks — Mapping the Human Brain](https://medium.com/predict/artificial-neural-networks-mapping-the-human-brain-2e0bd4a93160)\n",
    "\n",
    "[Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://www.researchgate.net/figure/Various-forms-of-non-linear-activation-functions-Figure-adopted-from-Caffe-Tutorial_fig3_315667264)\n",
    "\n",
    "[A Beginner's Guide to Neural Networks and Deep Learning](https://towardsdatascience.com/understanding-neural-networks-22b29755abd9)\n",
    "\n",
    "[Neural Representation of AND, OR, NOT, XOR and XNOR Logic Gates (Perceptron Algorithm)](https://medium.com/@stanleydukor/neural-representation-of-and-or-not-xor-and-xnor-logic-gates-perceptron-algorithm-b0275375fea1)\n",
    "\n",
    "[Selecting the best Machine Learning algorithm for your regression problem](https://towardsdatascience.com/selecting-the-best-machine-learning-algorithm-for-your-regression-problem-20c330bad4ef)\n",
    "\n",
    "[Introduction to Artificial Neural Networks](https://towardsdatascience.com/introduction-to-artificial-neural-networks-ann-1aea15775ef9)\n",
    "\n",
    "[What Are Overfitting and Underfitting in Machine Learning?](https://towardsdatascience.com/what-are-overfitting-and-underfitting-in-machine-learning-a96b30864690)\n",
    "\n",
    "[The Backpropagation Algorithm Demystified](https://www.kdnuggets.com/2019/01/backpropagation-algorithm-demystified.html)\n",
    "\n",
    "[Neuron](https://medicalxpress.com/news/2018-07-neuron-axons-spindly-theyre-optimizing.html)\n",
    "\n",
    "[Underfitting/Overfitting](https://in.pinterest.com/pin/672232681855858689/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[Home Page](../Start_Here.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;\n",
    "[Next Notebook](Part_2.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
