{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let's execute the cell below to display information about the CUDA driver and GPUs running on the server by running the `nvidia-smi` command. To do this, execute the cell block below by giving it focus (clicking on it with your mouse), and hitting Ctrl-Enter, or pressing the play button in the toolbar above. If all goes well, you should see some output returned below the grey cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "The **goal** of this lab is to:\n",
    "\n",
    "- Learn how to run the same code on both a multicore CPU and a GPU using pSTL library\n",
    "- Understand steps required to make a sequential code parallel using pSTL constructs\n",
    "\n",
    "We do not intend to cover:\n",
    "- Detailed optimization techniques and mapping of stdpar constructs to CUDA C\n",
    "\n",
    "\n",
    "# STL\n",
    "If you are not familiar with STL (Standard Teamplate Library), this section will give you a brief introduction that would be required to understand the usage of STL library for our code.\n",
    "\n",
    "The C++ STL (Standard Template Library) is a powerful set of C++ template classes to provide general-purpose classes and functions with templates that implement many popular and commonly used algorithms and data structures like vectors, lists, queues, and stacks.\n",
    "\n",
    "At the core of the C++ Standard Template Library are following three well-structured components \n",
    "\n",
    "- Containers: Containers are used to manage collections of objects of a certain kind. There are several different types of containers like deque, list, vector, map etc.\n",
    "\n",
    "- Algorithms: Algorithms act on containers. They provide the means by which you will perform initialization, sorting, searching, and transforming of the contents of containers.\n",
    "\n",
    "- Iterators: Iterators are used to step through the elements of collections of objects. These collections may be containers or subsets of containers.\n",
    "\n",
    "For our code to make *Pair Calculation* we will be making use of ```vector``` container. The example below will introduce you to the container and how to use iterator to step through elements of vector. ```vector``` container (a C++ Standard Template) which is similar to an array with an exception that it automatically handles its own storage requirements in case it grows\n",
    "\n",
    "For our code we will be making use of ```std::for_each``` alogithm and its sample usage is also shown in code below:\n",
    "\n",
    "```cpp\n",
    "#include <vector>\n",
    "#include <algorithm>\n",
    "#include <iostream>\n",
    " \n",
    "//Using functor\n",
    "struct Sum\n",
    "{\n",
    "    void operator()(int n) { sum += n; }\n",
    "    int sum{0};\n",
    "};\n",
    " \n",
    "int main()\n",
    "{\n",
    "    std::vector<int> nums{3, 4, 2, 8, 15, 267};\n",
    " \n",
    "    auto print = [](const int& n) { std::cout << \" \" << n; };\n",
    " \n",
    "    std::cout << \"before:\";\n",
    "    std::for_each(nums.cbegin(), nums.cend(), print);\n",
    "    std::cout << '\\n';\n",
    " \n",
    "    std::for_each(nums.begin(), nums.end(), [](int &n){ n++; });\n",
    " \n",
    "    // calls Sum::operator() for each number\n",
    "    Sum s = std::for_each(nums.begin(), nums.end(), Sum());\n",
    " \n",
    "    std::cout << \"after: \";\n",
    "    std::for_each(nums.cbegin(), nums.cend(), print);\n",
    "    std::cout << '\\n';\n",
    "    std::cout << \"sum: \" << s.sum << '\\n';\n",
    "}\n",
    "```\n",
    "\n",
    "To learn more about STL you can read and execute sample codes [here](https://www.tutorialspoint.com/cplusplus/cpp_stl_tutorial.htm).\n",
    "\n",
    "\n",
    "# Parallel STL\n",
    "Starting with C++17, parallelism has become an integral part of the standard itself. Parallel STL (pSTL) is an implementation of the C++ standard library algorithms with support for execution policies, commonly called C++17.\n",
    "\n",
    "C++17 Parallel Standard Library (stdpar) introduces parallel and vector concurrency for standard algorithms. It is important to note that stdpar is a library and not a language extension.\n",
    "\n",
    "\n",
    "## std::par Execution Policies\n",
    "\n",
    "\n",
    "Execution Policies define the kind of parallelism that will be applied to parallel algorithms. Most standard algorithms included in STL support execution policies. Defined below are the execution policies:\n",
    "\n",
    "- std::execution::seq = sequential\n",
    "    - This execution policy type used as a unique type to disambiguate parallel algorithm overloading and require that a parallel algorithm’s execution may not be parallelized.\n",
    "- std::execution::par = parallel\n",
    "    - This execution policy type used as a unique type to disambiguate parallel algorithm overloading and indicate that a parallel algorithm’s execution may be parallelized\n",
    "- std::execution::par_unseq = parallel + vectorized\n",
    "    - This execution policy type used as a unique type to disambiguate parallel algorithm overloading and indicate that a parallel algorithm’s execution may be parallelized and vectorized\n",
    "\n",
    "Implementation of execution policies is provided by different compilers from specific vendors. For GPU parallel execution policy we will be making use of NVIDIA compiler. \n",
    "\n",
    "\n",
    "## Historical Perspective\n",
    "\n",
    "Changes to how the call to _stl_ algorithms changed the new version of C++ standard to incorporate execution policies:\n",
    "\n",
    "**C++98:** \n",
    "```cpp\n",
    "std::sort(c.begin(), c.end()); \n",
    "```\n",
    "**C++17:** \n",
    "```cpp\n",
    "std::sort(std::execution::par, c.begin(), c.end());\n",
    "```\n",
    "\n",
    "We will be using PGI compiler provided pSTL implementation to compile and run code on NVIDIA GPU. Currently the PGI implementation of C++ parallel algorithms uses [Unified Memory](../GPU_Architecture_Terminologies.ipynb), which means that heap memory is automatically shared between a CPU(Host) and GPU(Device). Stack memory and global memory are not shared. Below given example shows the right allocation and usage of the pSTL.\n",
    "\n",
    "```cpp\n",
    "std::vector<int> v = ...;\n",
    "std::sort(std::execution::par, v.begin(), v.end()); // OK, vector allocates on heap\n",
    "\n",
    "std::array<int, 1024> a = ...;\n",
    "std::sort(std::execution::par, a.begin(), a.end()); // Fails, array stored on the stack\n",
    "```\n",
    "\n",
    "For our code we will be making use of ```std::for_each``` alogithm with support for ```std::execution::par``` execution policy\n",
    "\n",
    "**Counting Iterator**: In our code we will also be using a special iterator ```counting_iterator```. This iterator which represents a pointer into a range of sequentially changing values. This iterator is useful for creating a range filled with a sequence without explicitly storing it in memory. Using ```counting_iterator``` saves memory capacity and bandwidth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets start modifying the original code and add stdpar. From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/stdpar` directory. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile and Run for Multicore\n",
    "\n",
    "Now that we have added a pSTL code, let us try compile the code. We will be using NVIDIA HPC SDK for this exercise. The flags used for enabling parallel STL target offloading are as follows:\n",
    "\n",
    "`-stdpar` : This flag tell the compiler to enable Parallel STL for a respective target\n",
    "- `stdpar=multicore` will allow us to compile our code for a multicore\n",
    "- `stdpar` will allow us to compile our code for a NVIDIA GPU (Default is NVIDIA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Compile the code for muticore\n",
    "!cd ../../source_code/stdpar && nvc++ -std=c++17 -stdpar=multicore -o rdf rdf.cpp -I/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.0/include -ltbb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run the multicore code\n",
    "!cd ../../source_code/stdpar && ./rdf && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output entropy value should be the following:\n",
    "\n",
    "```\n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nsys\n",
    "!cd ../../source_code/stdpar && nsys profile -t nvtx --stats=true --force-overwrite true -o rdf_stdpar_multicore ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/stdpar/rdf_stdpar_multicore.qdrep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/stdpar_multicore.png\">\n",
    "\n",
    "\n",
    "### Compile and run for Nvidia GPU\n",
    "\n",
    "Without changing the code now let us try to recompile the code for NVIDIA GPU and rerun.\n",
    "GPU acceleration of C++ Parallel Algorithms is enabled with the `-⁠stdpar` command-line option to NVC++. If `-⁠stdpar `is specified, almost all algorithms that use a parallel execution policy are compiled for offloading to run in parallel on an NVIDIA GPU.\n",
    "\n",
    " **Understand and analyze** the solution present at:\n",
    "\n",
    "[RDF Code](../../source_code/stdpar/SOLUTION/rdf.cpp)\n",
    "\n",
    "[File Reader](../../source_code/stdpar/SOLUTION/dcdread.h)\n",
    "\n",
    "Open the downloaded files for inspection. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/stdpar && nvc++ -std=c++17 -DUSE_COUNTING_ITERATIOR  -stdpar=gpu -o rdf rdf.cpp "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run on NVIDIA GPU\n",
    "!cd ../../source_code/stdpar && ./rdf && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output entropy value should be the following:\n",
    "\n",
    "```\n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx\n",
    "!cd ../../source_code/stdpar && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_stdpar_gpu ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/stdpar/rdf_stdpar_gpu.qdrep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/stdpar_gpu.png\">\n",
    "\n",
    "If you inspect the output of the profiler closer, you can see the usage of *Unified Memory* annotated with green rectangle which was explained in previous sections.\n",
    "\n",
    "Moreover, if you compare the NVTX marker `Pair_Calculation` (from the NVTX row) in both multicore and GPU version, you can see how much improvement you achieved. In the *example screenshot*, we were able to reduce that range from 1.52 seconds to 188.4 mseconds.\n",
    "\n",
    "Feel free to checkout the [solution](../../source_code/stdpar/SOLUTION/rdf.cpp) to help you understand better or compare your implementation with the sample solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# stdpar Analysis\n",
    "\n",
    "**Usage Scenarios**\n",
    "- stdpar is part of the standard language and provides a good start for accelerating code on accelerators like GPU and multicores.\n",
    "\n",
    "**Limitations/Constraints**\n",
    "-  stdpar is available as part of C++17 onwards and does not work with other popular languages like Fortran which is supported by OpenMP and OpenACC.\n",
    "  \n",
    "\n",
    "**Which Compilers Support stdpar on GPUs and Multicore?**\n",
    "1. NVIDIA GPU: As of April 2020 the compiler that support std::par on NVIDIA GPU is from NVIDIA/PGI. \n",
    "2. x86 Multicore: gcc has an implementation on a multicore CPU which is based on Intel TBB in the backend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-Lab Summary\n",
    "\n",
    "If you would like to download this lab for later viewing, it is recommend you go to your browsers File menu (not the Jupyter notebook file menu) and save the complete web page.  This will ensure the images are copied down as well. You can also execute the following cell block to create a zip-file of the files you've been working on, and download it with the link below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ..\n",
    "rm -f nways_files.zip\n",
    "zip -r nways_files.zip *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After** executing the above zip command, you should be able to download the zip file [here](../nways_files.zip). Let us now go back to parallelizing our code using other approaches.\n",
    "\n",
    "**IMPORTANT**: Please click on **HOME** to go back to the main notebook for *N ways of GPU programming for MD* code.\n",
    "\n",
    "-----\n",
    "\n",
    "# <p style=\"text-align:center;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\"> <a href=../../../nways_MD_start.ipynb>HOME</a></p>\n",
    "\n",
    "-----\n",
    "\n",
    "\n",
    "# Links and Resources\n",
    "[stdpar Guide](https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9770-c++17-parallel-algorithms-for-nvidia-gpus-with-pgi-c++.pdf)\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "\n",
    "**NOTE**: To be able to see the Nsight System profiler output, please download Nsight System latest version from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [OpenACC Resources](https://www.openacc.org/resources) and join our [OpenACC Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- \n",
    "\n",
    "## Licensing \n",
    "\n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
