{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let's execute the cell below to display information about the CUDA driver and GPUs running on the server by running the `nvidia-smi` command. To do this, execute the cell block below by giving it focus (clicking on it with your mouse), and hitting Ctrl-Enter, or pressing the play button in the toolbar above. If all goes well, you should see some output returned below the grey cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "The **goal** of this lab is to:\n",
    "\n",
    "- Learn how to find bottleneck and performance limiters using Nsight tools\n",
    "- Learn about ways to to extract more parallelism in loops\n",
    "- Dig deeper into kernels by analyzing it with Nsight Compute\n",
    "\n",
    "\n",
    "In this section, we will optimize the parallel [RDF](../serial/rdf_overview.ipynb) application using OpenMP offloading. Before we begin, feel free to have a look at the parallel version of the code and inspect it once again. \n",
    "\n",
    "[RDF Parallel Code](../../source_code/openmp/SOLUTION/rdf_offload_split.cpp)\n",
    "\n",
    "[File Reader](../../source_code/openmp/SOLUTION/dcdread.h)\n",
    "\n",
    "Now, let's compile, and profile it with Nsight Systems first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openmp && nvc++ -mp=gpu -Minfo=mp -o rdf SOLUTION/rdf_offload.cpp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx\n",
    "!cd ../../source_code/openmp && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_offload ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openacc/rdf_offload.qdrep) and open it via the GUI. Then profile the specific kernel `pair_gpu` with Nsight compute. We use this as the base to compare with the next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile with nsight compute\n",
    "!cd ../../source_code/openmp && ncu --set full --launch-skip 0 --launch-count 1 -k pair_gpu -o rdf_offload ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openmp/rdf_offload.ncu-rep) and open it via the GUI.\n",
    "\n",
    "## Further Optimization\n",
    "Currently both our distributed and workshared parallelism comes from the same loop. In other words, in this example, we used teams and distribute to spread parallelism across the full GPU. As a result, all of the parallelism comes from the outer loop. There are multiple ways to increase parallelism: \n",
    "\n",
    "- Moving the `Parallel For` to the inner loop\n",
    "- Collapsing the loops together\n",
    "\n",
    "In the following sections, we will explore each of these methods.\n",
    "\n",
    "\n",
    "### Split Teams Distribute from Parallel For\n",
    "\n",
    "In this method, we distribute outer loop to thread teams and workshare inner loop across threads.The example below shows usage of OpenMP target splitting teams distribute from parallel for. The example code contains two levels of\n",
    "OpenMP offloading parallelism in a double-nested loop.\n",
    "\n",
    "```cpp\n",
    "#pragma omp target teams distribute \n",
    "    for( int j = 1; j < N; j++) {\n",
    "      #pragma parallel for \n",
    "      for( int i = 1; i < M; i++ ) {\n",
    "            <loop code>\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "Look at the profiler report from the previous section again. From the timeline, have a close look at the the kernel functions. Checkout the theoretical occupancy. As shown in the example screenshot below, `pair_gpu_182` kernel has the theoretical occupancy of 50%. It clearly shows that the occupancy is a limiting factor. *Occupancy* is a measure of how well the GPU compute resources are being utilized. It is about how much parallelism is running / how much parallelism the hardware could run.\n",
    "\n",
    "<img src=\"../images/data_thread.png\">\n",
    "\n",
    "NVIDIA GPUs are comprised of multiple streaming multiprocessors (SMs) where it can manage up to 2048 concurrent threads (not actively running at the same time). Low occupancy shows that there are not enough active threads to fully utilize the computing resources. Higher occupancy implies that the scheduler has more active threads to choose from and hence achieves higher performance. So, what does this mean in OpenACC execution model?\n",
    "\n",
    "\n",
    "\n",
    "Now, lets start modifying the original code and split the teams distribute. From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/openmp` directory. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openmp && nvc++ -mp=gpu -Minfo=mp -o rdf rdf.cpp "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the compiler feedback (you should get a similar output as below) you can see from *Line 172* and *Line 178* that it separated the teams distribute from parallel for.\n",
    "\n",
    "<img src=\"../images/openmp_feedback_offload_split.png\">\n",
    "\n",
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run on Nvidia GPU and check the output\n",
    "!cd ../../source_code/openmp && ./rdf && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output should be the following:\n",
    "\n",
    "```\n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx\n",
    "!cd ../../source_code/openmp && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_offload_split ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openmp/rdf_offload_split.qdrep) and open it via the GUI. Compare the execution time for the `Pair_Calculation` from the NVTX row with the previous section and see how much the performance was improved.\n",
    "\n",
    "Let's dig deeper and profile the kernel with Nsight Compute and compare it with the base version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile with nsight compute\n",
    "!cd ../../source_code/openmp && ncu --set full --launch-skip 0 --launch-count 1 -k pair_gpu -o rdf_offload_split ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout Nsight compute report. [Download the profiler output](../../source_code/openmp/rdf_offload_split.ncu-rep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/compute_split.png\">\n",
    "\n",
    "When compared to the base version, we achieved 41% improvement in SM utilization and 358% utilization in memory.\n",
    "\n",
    "Feel free to checkout the [solution](../../source_code/openmp/SOLUTION/rdf_offload_split.cpp) to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collapse clause\n",
    "\n",
    "Specifying the `collapse(n)` clause takes the next `n` tightly-nested loops, folds them into one, and applies the OpenMP directives to the new loop. Collapsing loops means that two loops of trip counts N and M respectively will be automatically turned into a single loop with a trip count of N times M. By collapsing two or more parallel loops into a single loop the compiler has an increased amount of parallelism to use when mapping the code to the device. On highly parallel architectures, such as GPUs, this will give us more parallelism to distribute and better performance.\n",
    "\n",
    "Try using the collapse clause and observe any performance difference. How much this optimization will speed-up the code will vary according to the application and the target accelerator, but it is not uncommon to see large speed-ups by using collapse on loop nests.\n",
    "\n",
    "In the example below, we collapse the two loops before applying both teams and thread parallelism to both.\n",
    "\n",
    "```cpp\n",
    "#pragma omp target teams distribute parallel for collapse(2) \n",
    "for (int i = 0; i < N; i++ )\n",
    "{\n",
    "    for (int j=0;j< M;j++)\n",
    "        < loop code >\n",
    "} \n",
    "```\n",
    "\n",
    "Now, lets start modifying the original code and add the collapse clause. From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/openmp` directory. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openmp && nvc++ -mp=gpu -Minfo=mp -o rdf rdf.cpp "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run on Nvidia GPU and check the output\n",
    "!cd ../../source_code/openmp && ./rdf && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output should be the following:\n",
    "\n",
    "```\n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx\n",
    "!cd ../../source_code/openmp && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_collapse ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openmp/rdf_collapse.qdrep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/openmp_gpu_collapse.png\">\n",
    "\n",
    "Compare the execution time for the `Pair_Calculation` from the NVTX row (annotated in Red rectangle in the example screenshot) with the previous section. It is clear the using collapse clause improved the performance by extracting more parallelism. Please note when you compare the two methods explored here, one may give better results than the other.\n",
    "\n",
    "Let's dig deeper and profile the kernel with Nsight Compute and compare it with the base version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile with nsight compute\n",
    "!cd ../../source_code/openmp && ncu --set full --launch-skip 0 --launch-count 1 -k pair_gpu -o rdf_collapse ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the Nsight compute report. [Download the profiler output](../../source_code/openmp/rdf_collapse.ncu-rep) and open it via the GUI. Have a look at the example expected profiler report below (yellow line is the base version, blue is the current):\n",
    "\n",
    "<img src=\"../images/compute_collapse.png\">\n",
    "\n",
    "When compared to the base version (using baseline feature on Nsight compute), we achieved 128% improvement in SM utilization and 66% utilization in memory. We can also see the improvement and better utilization in the roofline analysis (see example screenshot below).\n",
    "\n",
    "<img src=\"../images/compute_collapse_roofline.png\">\n",
    "\n",
    "Dots with the blue outline is the optimzied version and the ones with the orange outline is the base version.\n",
    "\n",
    "Feel free to checkout the [solution](../../source_code/openmp/SOLUTION/rdf_offload_collapse.cpp) to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `num_teams(n)` and `num_threads`\n",
    "\n",
    "When using the `distribute parallel for` construct, the for loop is distributed across all threads for all teams of the current teams region. For example, if there are 10 teams, and each team consists of 256 threads, the loop will be distributed across 2560 threads. You can explicitly specify the number of threads to be created in the team using the `num_threads(m)`. Moreover, you can specify the maximum number of teams created by using `num_teams(n)` (the actual number of teams may be smalled than this number). You can add these attributes to the `collpase(x)` clause or `parallel for` clause. \n",
    "\n",
    "\n",
    "```cpp\n",
    "#pragma omp target teams distribute parallel for collapse(2) num_threads(256)\n",
    "for (int i = 0; i < N; i++ )\n",
    "{\n",
    "    for (int j=0;j< M;j++)\n",
    "        < loop code >\n",
    "} \n",
    "```\n",
    "\n",
    "Now, lets start modifying the original code and add `num_threads(n)` to the `collapse` clause. From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/openmp` directory. Remember to **SAVE** your code after changes, before running below cells.\n",
    "\n",
    "**NOTE:** To do some more experiment, try changing the value of `n` to 128, 256, 512. Then compile, run and profile the code and compare the result. Are there any differences? What does the profiler show?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openmp && nvc++ -mp=gpu -Minfo=mp -o rdf rdf.cpp "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to validate the output by running the executable and validate the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run on Nvidia GPU and check the output\n",
    "!cd ../../source_code/openmp && ./rdf && cat Pair_entropy.dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output should be the following:\n",
    "\n",
    "```\n",
    "s2 value is -2.43191\n",
    "s2bond value is -3.87014\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output of nvptx\n",
    "!cd ../../source_code/openmp && nsys profile -t nvtx,cuda --stats=true --force-overwrite true -o rdf_offload_num ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openmp/rdf_offload_num.qdrep) and open it via the GUI. Let's dig deeper and profile the `pair_gpu` kernel with Nsight compute and compare it with the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile with nsight compute\n",
    "!cd ../../source_code/openmp && ncu --set full --launch-skip 0 --launch-count 1 -k pair_gpu -o rdf_collapse_num ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the Nsight compute report. [Download the profiler output](../../source_code/openmp/rdf_collapse_num.ncu-rep) and open it via the GUI. Feel free to checkout the example [solution](../../source_code/openmp/SOLUTION/rdf_offload_collapse_num.cpp) to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-Lab Summary\n",
    "\n",
    "If you would like to download this lab for later viewing, it is recommend you go to your browsers File menu (not the Jupyter notebook file menu) and save the complete web page.  This will ensure the images are copied down as well. You can also execute the following cell block to create a zip-file of the files you've been working on, and download it with the link below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ..\n",
    "rm -f nways_files.zip\n",
    "zip -r nways_files.zip *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After** executing the above zip command, you should be able to download the zip file [here](../nways_files.zip). Let us now go back to parallelizing our code using other approaches.\n",
    "\n",
    "**IMPORTANT**: Please click on **HOME** to go back to the main notebook for *N ways of GPU programming for MD* code.\n",
    "\n",
    "-----\n",
    "\n",
    "# <p style=\"text-align:center;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\"> <a href=../../../nways_MD_start.ipynb>HOME</a></p>\n",
    "\n",
    "-----\n",
    "\n",
    "\n",
    "# Links and Resources\n",
    "[OpenMP Programming Model](https://computing.llnl.gov/tutorials/openMP/)\n",
    "\n",
    "[OpenMP Target Directive](https://www.openmp.org/wp-content/uploads/openmp-examples-4.5.0.pdf)\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "[NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute)\n",
    "\n",
    "**NOTE**: To be able to see the Nsight System profiler output, please download Nsight System latest version from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [OpenACC Resources](https://www.openacc.org/resources) and join our [OpenACC Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- \n",
    "\n",
    "## Licensing \n",
    "\n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
