{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Megatron GPT Bootcamp\n",
    "\n",
    "## Learning objectives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this bootcamp is designed for training very large language models with NVIDIA [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) in a step-wised manner. \n",
    "\n",
    "There are two labs, each with a focus point. \n",
    "\n",
    "In Lab 1, we will learn the default [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) workflow, highlighting :\n",
    "\n",
    "   - How to calculate time-to-compute needs for resource planning.\n",
    "    \n",
    "   - Understanding Megatron-LM's core engine - Model Parallel Unit(MPU)\n",
    "    \n",
    "   - Profiling : core concepts on GPUs performance across multi-gpus and/or multi-node runs.\n",
    "\n",
    "In Lab 2, the focus is shifted to the **customization** of [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) workflow. We will walk through and exercise steps for customization of the Megatron-LM's workflow in order to address to local langauge needs.  \n",
    "\n",
    "\n",
    "* Standard: Python\n",
    "* Frameworks: Pytorch + Megatron-LM \n",
    "\n",
    "It is required to have more than one GPU for this bootcamp.\n",
    "\n",
    "This bootcamp is tested on 2 x A100 GPUS with 40G memory. One should also have [NVLink / NVSwitch](https://www.nvidia.com/en-in/data-center/nvlink/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start by checking available gpus in the environment using nvidia-smi "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify you have 2 x A100 GPUs, each with 40G memory, below is an example of expected outputs : \n",
    "\n",
    "            Wed Sep 15 09:14:15 2021       \n",
    "            +-----------------------------------------------------------------------------+\n",
    "            | NVIDIA-SMI 460.27.04    Driver Version: 460.27.04    CUDA Version: 11.2     |\n",
    "            |-------------------------------+----------------------+----------------------+\n",
    "            | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
    "            | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
    "            |                               |                      |               MIG M. |\n",
    "            |===============================+======================+======================|\n",
    "            |   0  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |\n",
    "            | N/A   24C    P0    57W / 400W |      0MiB / 40536MiB |      4%      Default |\n",
    "            |                               |                      |             Disabled |\n",
    "            +-------------------------------+----------------------+----------------------+\n",
    "            |   1  A100-SXM4-40GB      On   | 00000000:0F:00.0 Off |                    0 |\n",
    "            | N/A   24C    P0    53W / 400W |      0MiB / 40536MiB |      0%      Default |\n",
    "            |                               |                      |             Disabled |\n",
    "            +-------------------------------+----------------------+----------------------+\n",
    "\n",
    "            +-----------------------------------------------------------------------------+\n",
    "            | Processes:                                                                  |\n",
    "            |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
    "            |        ID   ID                                                   Usage      |\n",
    "            |=============================================================================|\n",
    "            |  No running processes found                                                 |\n",
    "            +-----------------------------------------------------------------------------+\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# verify nvlink status\n",
    "!nvidia-smi nvlink --status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify NVlink is active, below is an example of expected outputs : \n",
    "\n",
    "        GPU 0: A100-SXM4-40GB (UUID: GPU-2e4d2105-718d-3b94-6f0f-25c148681e83)\n",
    "             Link 0: 25 GB/s\n",
    "             Link 1: 25 GB/s\n",
    "             Link 2: 25 GB/s\n",
    "             Link 3: 25 GB/s\n",
    "             Link 4: 25 GB/s\n",
    "             Link 5: 25 GB/s\n",
    "             Link 6: 25 GB/s\n",
    "             Link 7: 25 GB/s\n",
    "             Link 8: 25 GB/s\n",
    "             Link 9: 25 GB/s\n",
    "             Link 10: 25 GB/s\n",
    "             Link 11: 25 GB/s\n",
    "        GPU 1: A100-SXM4-40GB (UUID: GPU-49615223-919e-6f9f-ad79-69d86bc1a13b)\n",
    "             Link 0: 25 GB/s\n",
    "             Link 1: 25 GB/s\n",
    "             Link 2: 25 GB/s\n",
    "             Link 3: 25 GB/s\n",
    "             Link 4: 25 GB/s\n",
    "             Link 5: 25 GB/s\n",
    "             Link 6: 25 GB/s\n",
    "             Link 7: 25 GB/s\n",
    "             Link 8: 25 GB/s\n",
    "             Link 9: 25 GB/s\n",
    "             Link 10: 25 GB/s\n",
    "             Link 11: 25 GB/s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# verify profiling capacility \n",
    "!nsys status -e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Verify Profiling Capability, the expected output should look something simialr to the below\n",
    "\n",
    "            Sampling Environment Check\n",
    "            Linux Kernel Paranoid Level = 2: OK\n",
    "            Linux Distribution = Ubuntu\n",
    "            Linux Kernel Version = 4.18.0-305.12.1.el8_4.x86_64: OK\n",
    "            Linux perf_event_open syscall available: OK\n",
    "            Sampling trigger event available: OK\n",
    "            Intel(c) Last Branch Record support: Not Available\n",
    "            Sampling Environment: OK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "To start with, we need to create placeholder for dataset. We are going to populate these folders later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.makedirs('./dataset/EN/32k', exist_ok=True)\n",
    "os.makedirs('./dataset/EN/50k', exist_ok=True)\n",
    "os.makedirs('./dataset/SV/32k', exist_ok=True)\n",
    "os.makedirs('./dataset/SV/56k', exist_ok=True)\n",
    "os.makedirs('./sv_ckpt/', exist_ok=True)\n",
    "os.makedirs('./profiles/naive', exist_ok=True)\n",
    "os.makedirs('./profiles/2ndrun', exist_ok=True)\n",
    "os.makedirs('./profiles/SV', exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Tutorial Outline\n",
    "\n",
    "The following contents will be covered during the Bootcamp :\n",
    "\n",
    "- **Outlines of Lab 1**\n",
    "    Megatron 101 in half a day - Please go through the below notebooks sequentially.\n",
    "    1. [WebCrawling](./jupyter_notebook/Megatron-LM/tools/openwebtext/Lab1-1_Website_scrapping.ipynb)\n",
    "    2. [Estimate hours/days needed to execute one end-to-end run per Megatron configuration](./jupyter_notebook/Lab1-2_EstimateComputeDaysNeeded.ipynb)\n",
    "    3. [Understanding the core of Megatron - mpu ](./jupyter_notebook/Lab1-3_MegatronFundementals.ipynb)\n",
    "    4. [About GPT's tokenizer](./jupyter_notebook/Lab1-4_GPT_vocab_merge_files.ipynb)\n",
    "    5. [jsonfy and convert to mmap format](./jupyter_notebook/Lab1-5_jsonfy_and_process2mmap.ipynb)\n",
    "    6. [Megatron runs vs config](./jupyter_notebook/Lab1-6_Observe_GPT_runs_vs_performance.ipynb)\n",
    "\n",
    "- **Outlines of Day 3**\n",
    "    Getting started on training own language Megatron GPT models -- Please go through the below notebooks sequentially.\n",
    "    1. [Fetch and extract Swedish data](./jupyter_notebook/Megatron-LM/tools/openwebtext/Lab2-1_acquiring_data.ipynb)\n",
    "    2. [Find sentence boundary and deduplicate your data](./jupyter_notebook/Megatron-LM/tools/openwebtext/Lab2-2_SentenceBoundary_and_Deduplicate.ipynb)\n",
    "        - [mini challenge - approaching groundtruth](./jupyter_notebook/Megatron-LM/tools/openwebtext/Lab2-2_SentenceBoundary_and_Deduplicate.ipynb#TheChallenge)\n",
    "    3. [Train your own GPTBPE Tokenizer on your own data ](./jupyter_notebook/Lab2-3_train_own_GPT2BPETokenizer.ipynb)\n",
    "    4. [customize preprocess data python script and convert to mmap](./jupyter_notebook/Lab2-4_customize_process2mmap.ipynb)\n",
    "    5. [The Challenge - Go Big or go home!](./jupyter_notebook/Lab2-5_run_Megatron_with_varying_config.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tutorial Duration\n",
    "The lab material will be presented in a 8 hr session. Link to material is available for download at the end of the gpubootcamp. \n",
    "\n",
    "### Content Level\n",
    "Intermediate , Advanced\n",
    "\n",
    "### Target Audience and Prerequisites\n",
    "The target audience for this lab is researchers/graduate students and developers who are interested in learning about scaling their Deep learning systems to multiple GPUs to accelerate their scientific applications.\n",
    "\n",
    "Basic understanding on Deep learning is required, If you are new to Deep learning , it is recommended to go through the [Distributed_Deep_Learning bootcamp](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/ai/Distributed_Deep_Learning/English/python) prior.\n",
    " \n",
    "**Disclaimer** : All the results mentioned in the notebooks were tested on a *DGX-1 machine equipped with 2 or 4 or 8 x Tesla V100 connected via NVLink*. The results would vary when using different hardware and would also depend on the Interconnect bandwidth and the thermal conditions of the machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "## Licensing\n",
    "\n",
    "This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}