{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Deep Learning Bootcamp\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "The objective of this bootcamp is to give an introduction to Distributed Deep Learning. This bootcamp will introduce participants to fundamentals of Distributed deep learning and give a hands-on experience on methods that can be applied to Deep learning models for faster model training.The Bootcamp assumes familiarity with Deep learning fundamentals and if you are new to Deep Learning , kindly go through the [AI for Climate Bootcamp](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc_ai/ai_science_climate) prior.\n",
    "\n",
    "* Standard: Python\n",
    "* Frameworks: Horovod , Tensorflow \n",
    "\n",
    "It is required to have more than one GPU for the bootcamp and we recommend using a [DGX](https://www.nvidia.com/en-in/data-center/dgx-systems/) like cluster with [NVLink / NVSwitch](https://www.nvidia.com/en-in/data-center/nvlink/) support.\n",
    "\n",
    "Let's start with testing the GPUs you are running the code on in this bootcamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tutorial Outline\n",
    "\n",
    "The following contents will be covered during the Bootcamp :\n",
    "\n",
    "- [**Introduction to Distributed deep learning**](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb)\n",
    "    - [The need for Distributed Deep Learning](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#The-need-for-Distributed-Deep-Learning)\n",
    "    - [Differnet types of Distributed Deep learning and it's applications](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#Differnet-types-of-Distributed-Deep-learning-and-it's-applications)\n",
    "        - [Training and Inference](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#Training-and-Inference)\n",
    "        - [Data and Model Parallelism](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#Data-and-Model-Parallelism)\n",
    "        - [Framework and NVIDIA NGC Support - Optional](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#Framework-and-NVIDIA-NGC-Support---Optional)\n",
    "    - [Demo - Scalability across multiple GPUs](jupyter_notebook/1.Introduction-to-Distributed-Deep-Learning.ipynb#Demo---Scalability-across-multiple-GPUs) \n",
    "\n",
    "\n",
    "- [**System Topology**](jupyter_notebook/2.1.System-Topology.ipynb)\n",
    "    - [Understanding System Topology](jupyter_notebook/2.1.System-Topology.ipynb#Understanding-System-Topology)\n",
    "        - [Communication concepts](jupyter_notebook/2.1.System-Topology.ipynb#Communication-concepts)\n",
    "    - [Intra-Node Communication Topology](jupyter_notebook/2.1.System-Topology.ipynb#Intra-Node-communication-Topology)\n",
    "        - [Performance variation due to system topology](jupyter_notebook/2.1.System-Topology.ipynb#Performance-variation-due-to-system-topology)\n",
    "        - [Profiling using DLProf](jupyter_notebook/2.1.System-Topology.ipynb#Profiling-using-DLProf)\n",
    "    - [NCCL](jupyter_notebook/2.1.System-Topology.ipynb#NCCL)\n",
    "        - [NCCL_P2P_LEVEL=0 or P2P Disabled](jupyter_notebook/2.1.System-Topology.ipynb#NCCL_P2P_LEVEL=0-or-P2P-Disabled)\n",
    "        - [NCCL_P2P_LEVEL=1 or P2P via PCIe](jupyter_notebook/2.1.System-Topology.ipynb#NCCL_P2P_LEVEL=1-or-P2P-via-PCIe)\n",
    "    - [Benchmarking the system topology](jupyter_notebook/2.1.System-Topology.ipynb#Benchmarking-the-system-topology)\n",
    "\n",
    "\n",
    "- [**Hands-on with Distributed training**](jupyter_notebook/3.Hands-on-Multi-GPU.ipynb)\n",
    "    - [Tensorflow - Keras](jupyter_notebook/3.Hands-on-Multi-GPU.ipynb#Tensorflow---Keras)\n",
    "    - [Horovod](jupyter_notebook/3.Hands-on-Multi-GPU.ipynb#Horovod)\n",
    "\n",
    "\n",
    "\n",
    "- [**Challenges with convergence**](jupyter_notebook/4.Convergence.ipynb)\n",
    "    - [Concepts](jupyter_notebook/4.Convergence.ipynb#Concepts)\n",
    "        - [Impact of Batch size](jupyter_notebook/4.Convergence.ipynb#Impact-of-Batch-size)\n",
    "        - [Impact on test and validation accuracy](jupyter_notebook/4.Convergence.ipynb#Impact-on-test-and-validation-accuracy)\n",
    "    - [Techniques for faster convergence](jupyter_notebook/4.Convergence.ipynb#Techniques-for-faster-convergence)\n",
    "        - [Batch norm](jupyter_notebook/4.Convergence.ipynb#Batch-norm)\n",
    "        - [Learning rate scaling](jupyter_notebook/4.Convergence.ipynb#Learning-rate-scaling)\n",
    "        - [Learning rate warmup](jupyter_notebook/4.Convergence.ipynb#Learning-rate-warmup)\n",
    "        - [Using Optimizers built for Exascale Deep learning](jupyter_notebook/4.Convergence.ipynb#Using-Optimizers-built-for-Exascale-Deep-learning)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tutorial Duration\n",
    "The lab material will be presented in a 6hr session. Link to material is available for download at the end of the lab.\n",
    "\n",
    "### Content Level\n",
    "Beginner, Intermediate\n",
    "\n",
    "### Target Audience and Prerequisites\n",
    "The target audience for this lab is researchers/graduate students and developers who are interested in learning about scaling their Deep learning systems to multiple GPUs to accelerate their scientific applications.\n",
    "\n",
    "Basic understanding on Deep learning is required, If you are new to Deep learning , it is recommended to go through the [AI for Climate Bootcamp](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc_ai/ai_science_climate) prior.\n",
    " \n",
    "**Disclaimer** : All the results mentioned in the notebooks were tested on a *DGX-1 machine equipped with 8 x Tesla V100 connected via NVLink*. The results would vary when using different hardware and would also depend on the Interconnect bandwidth and the thermal conditions of the machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## Licensing\n",
    "\n",
    "This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}