{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin, let's execute the cell below to display information about the CUDA driver and GPUs running on the server by running the `nvidia-smi` command. To do this, execute the cell block below by giving it focus (clicking on it with your mouse), and hitting Ctrl-Enter, or pressing the play button in the toolbar above. If all goes well, you should see some output returned below the grey cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "The **goal** of this lab is to:\n",
    "\n",
    "- Learn how to find bottleneck and performance limiters using Nsight tools\n",
    "- Learn about three levels of parallelism in OpenACC\n",
    "- Learn how to use clauses to extract more parallelism in loops\n",
    "- Dig deeper into kernels by analyzing it with Nsight Compute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will optimize the parallel [RDF](../serial/rdf_overview.ipynb) application using OpenACC. Before we begin, feel free to have a look at the parallel version of the code and inspect it once again. \n",
    "\n",
    "[RDF Parallel Code](../../source_code/openacc/SOLUTION/rdf_data_directive.cpp)\n",
    "\n",
    "[File Reader](../../source_code/openacc/SOLUTION/dcdread.h)\n",
    "\n",
    "Now, let's compile, and profile it with Nsight Systems first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile the parallel code for Tesla GPU\n",
    "!cd ../../source_code/openacc && nvc++ -acc -ta=tesla:lineinfo  -Minfo=accel -o rdf SOLUTION/rdf_data_directive.cpp "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output\n",
    "!cd ../../source_code/openacc && nsys profile -t nvtx,openacc --stats=true --force-overwrite true -o rdf_parallel ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openacc/rdf_parallel.qdrep) and open it via the GUI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Optimization\n",
    "\n",
    "Look at the profiler report from the previous section again. From the timeline, have a close look at the the kernel functions. Checkout the theoretical occupancy. As shown in the example screenshot below, `pair_gpu_182` kernel has the theoretical occupancy of 50%. It clearly shows that the occupancy is a limiting factor. *Occupancy* is a measure of how well the GPU compute resources are being utilized. It is about how much parallelism is running / how much parallelism the hardware could run.\n",
    "\n",
    "<img src=\"../images/data_thread.png\">\n",
    "\n",
    "NVIDIA GPUs are comprised of multiple streaming multiprocessors (SMs) where it can manage up to 2048 concurrent threads (not actively running at the same time). Low occupancy shows that there are not enough active threads to fully utilize the computing resources. Higher occupancy implies that the scheduler has more active threads to choose from and hence achieves higher performance. So, what does this mean in OpenACC execution model?\n",
    "\n",
    "**3 Levels of Parallelism: Gang, Worker, and Vector**\n",
    "CUDA and OpenACC programming model use different terminologies for similar ideas. For example, in CUDA, parallel execution is organized into grids, blocks, and threads. On the other hand, the OpenACC execution model has three levels of *Vector*, *Worker* and *Gang*. *Vector* threads work in lockstep, performing a single operation on multipler data (SIMD), whereas *Workers* computes one vector and *Gangs* have 1 or more workers and they all share resources such as cache, or SMs. Gangs run independent of each other.\n",
    "\n",
    "\n",
    "OpenACC assumes the device has multiple processing elements (Streaming Multiprocessors on NVIDIA GPUs) running in parallel and mapping of OpenACC execution model on CUDA is as below:\n",
    "\n",
    "- An OpenACC gang is a threadblock\n",
    "- A worker is a warp\n",
    "- An OpenACC vector is a CUDA thread\n",
    "\n",
    "<img src=\"../images/diagram.png\" width=\"50%\" height=\"50%\">\n",
    "\n",
    "### Vector, Worker  and Gang Clauses\n",
    "\n",
    "So, in order to improve the occupancy, we have to increase the parallelism within the gang. In other words, we have to increase the number of threads that can be scheduled on the GPU to improve GPU thread occupancy.\n",
    "\n",
    "As you can see from the profiler report's screenshot, the grid dimension is fairly small `<53,1,1>` which shows the small amount of parallelism within the gang. We can use specific clauses to control the level of parallelism so that the compiler would use to parallelise the next loop. Each of *Vector*, *Worker* and *Gang* clauses can take a parameter to specify the size of each level of parallelism. For example we can control the vector length by using the `vector_length (num)` clause or we can add more workers by using `num_workers(num)`.\n",
    "\n",
    "```cpp\n",
    "#pragma acc parallel vector_length(32)\n",
    "#pragma acc loop gang num_workers(32)\n",
    "for (int i = 0; i < N; i++)\n",
    "    #pragma acc loop vector\n",
    "    for (int j = 0; j < N; j++) \n",
    "        ...\n",
    "```\n",
    "\n",
    "Now, add `gang`, `vector` and/or `worker` clauses to the code and experiment with the number of vectors and workers. Make necessary changes to the loop directives. Once done, save the file, re-compile via `make`, and profile it again. \n",
    "\n",
    "From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/openacc` directory. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openacc && make clean && make"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us start inspecting the compiler feedback and see if it applied the optimizations. Below is the screenshot of expected compiler feedback after adding the `gang`and `vector` clause to the code with vector length of 128. You can also change the vector length to 32 or 256 and see how the profiler output changes. The line 187 would change to `187, #pragma acc loop vector(256) /* threadIdx.x */` or `187, #pragma acc loop vector(32) /* threadIdx.x */`.\n",
    "\n",
    "<img src=\"../images/gang_vector.png\">\n",
    "\n",
    "Now, **Profile** your code with Nsight Systems command line `nsys`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output\n",
    "!cd ../../source_code/openacc && nsys profile -t nvtx,openacc --stats=true --force-overwrite true -o rdf_gang_vector ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openacc/rdf_gang_vector.qdrep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/gang_128.png\">\n",
    "\n",
    "Checkout the kernel functions on the timeline and the occupancy. As you can see from the above screenshot, the theoretical occupancy is now 75%. If we change the vector length to 32, the occupancy would be 50% as we do not have a lot of concurrent threads running.\n",
    "\n",
    "<img src=\"../images/gang_32.png\" width=\"30%\" height=\"30%\">\n",
    "\n",
    "The loop iteration count inside the `pair_gpu_182` function (line 184 according the above compiler feedback) is `numatm = 6720`. In this example, this number is the grid dimension of `<6720,1,1>`.\n",
    "\n",
    "\n",
    "How much this optimization will speed-up the code will vary according to the application and the target accelerator, but it is not uncommon to see large speed-ups by using collapse on loop nests. \n",
    "\n",
    "Feel free to checkout the [solution](../../source_code/openacc/SOLUTION/rdf_gang_vector_length.cpp) to help you understand better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collapse Clauses\n",
    "In order to expose more parallelism and improve the occupancy, we can use an additional clause called `collapse` in the `#pragma acc loop` to optimize loops. The loop directive gives the compiler additional information about the next loop in the source code through several clauses. Apply the `collapse(N)` clause to a loop directive to collapse the next `N` tightly-nested loops to be collapsed into a single, flattened loop. This is useful if you have many nested loops or when you have really short loops. Sample usage of collapse clause is given as follows:\n",
    "\n",
    "```cpp\n",
    "#pragma acc parallel loop collapse (2)\n",
    "for (int i = 0; i < N; i++ )\n",
    "{\n",
    "    for (int j = 0; j < N; j++ )\n",
    "    {\n",
    "        < loop code >\n",
    "    }\n",
    "} \n",
    "```\n",
    "\n",
    "When the loop count in any of some tightly nested loops is relatively small compared to the available number of threads in the device, creating a single iteration space across all the nested loops, increases the iteration count thus allowing the compiler to extract more parallelism.\n",
    "\n",
    "**Tips on where to use:**\n",
    "- Collapse outer loops to enable creating more gangs.\n",
    "- Collapse inner loops to enable longer vector lengths.\n",
    "- Collapse all loops, when possible, to do both\n",
    "\n",
    "Now, add `collapse` clause to the code and make necessary changes to the loop directives. Once done, save the file, re-compile via `make`, and profile it again. \n",
    "\n",
    "From the top menu, click on *File*, and *Open* `rdf.cpp` and `dcdread.h` from the current directory at `C/source_code/openacc` directory. Remember to **SAVE** your code after changes, before running below cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile for Tesla GPU\n",
    "!cd ../../source_code/openacc && make clean && make"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us start inspecting the compiler feedback and see if it applied the optimizations. Below is the screenshot of expected compiler feedback after adding the `collapse`clause to the code. You can see that nested loops on line 184 has been successfully collapsed.\n",
    "\n",
    "<img src=\"../images/collapse_feedback.png\">\n",
    "\n",
    "Now, **Profile** your code with Nsight Systems command line `nsys`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile and see output\n",
    "!cd ../../source_code/openacc && nsys profile -t nvtx,openacc --stats=true --force-overwrite true -o rdf_collapse ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openacc/rdf_collapse.qdrep) and open it via the GUI. Have a look at the example expected profiler report below:\n",
    "\n",
    "<img src=\"../images/collapse_thread.png\">\n",
    "\n",
    "Checkout the kernel functions on the timeline and the occupancy. As you can see from the above screenshot, the theoretical occupancy is now 75%.\n",
    "\n",
    "The iteration count for the collapsed loop is `numatm * numatm` where `numatm = 6720`  (in this example). So, the iteration count for this particular loop (collapse loop) inside the `pair_gpu_182` function is 45158400 and this number divided by the vector length of *128* is 352800. The maximum grid size is 65535 blocks in each dimension and we can see that we have a grid dimension of `<65535,1,1>` in this example.\n",
    "\n",
    "By creating a single iteration space across the nested loops and increasing the iteration count, we improved the occupancy and extracted more parallelism.\n",
    "\n",
    "**Notes:**\n",
    "- 100% occupancy is not required for, nor does it guarantee best performance.\n",
    "- Less than 50% occupancy is often a red flag\n",
    "\n",
    "How much this optimization will speed-up the code will vary according to the application and the target accelerator, but it is not uncommon to see large speed-ups by using collapse on loop nests. \n",
    "\n",
    "Feel free to checkout the [solution](../../source_code/openacc/SOLUTION/rdf_collapse.cpp) to help you understand better.\n",
    "\n",
    "\n",
    "There is another clause which may be useful in optimizing loops. *Tile* clause breaks down the next loops into tiles before parallelising and it promotes data locality as the device can then use data from nearby tiles. \n",
    "\n",
    "```cpp\n",
    "#pragma acc parallel loop tile (4,4)\n",
    "for (int i = 0; i < N; i++ )\n",
    "{\n",
    "    for (int j = 0; j < N; j++ )\n",
    "    {\n",
    "        < loop code >\n",
    "    }\n",
    "} \n",
    "```\n",
    "\n",
    "We did not cover this in the labs but this is something you can explore and compare with the explained methods of loop optimization.\n",
    "\n",
    "Now, let's dig deeper and profile the kernel with the Nsight Compute profiler. To do this, let's use the [solution](../../source_code/openacc/SOLUTION/rdf_collapse.cpp) as a reference to get a similar report from Nsight Compute. Run the application, and profile it with the Nsight Systems first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile the solution for Tesla GPU\n",
    "!cd ../../source_code/openacc && nvc++ -acc -ta=tesla:lineinfo  -Minfo=accel -o rdf SOLUTION/rdf_collapse.cpp "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile the solution with Nsight Systems \n",
    "!cd ../../source_code/openacc && nsys profile -t nvtx,openacc --stats=true --force-overwrite true -o rdf_collapse_solution ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the profiler's report. [Download the profiler output](../../source_code/openacc/rdf_collapse_solution.qdrep) and open it via the GUI. Now, right click on the kernel `pair_gpu_182_gpu` and click on \"Analyze the Selected Kernel with NVIDIA Nsight Compute\" (see below screenshot). \n",
    "\n",
    "<img src=\"../images/compute_analyz.png\">\n",
    "\n",
    "Then, make sure to tick the radio button next to \"Display the command line to user NVIDIA Nsight Compute CLI\". \n",
    "\n",
    "<img src=\"../images/compute_command_line.png\" width=\"50%\" height=\"50%\">\n",
    "\n",
    "Then, you simply copy the command, run it and analyze the selected kernel. \n",
    "\n",
    "<img src=\"../images/compute_command.png\" width=\"50%\" height=\"50%\">\n",
    "\n",
    "To profile the selected kernel, run the below cell (by adding `--set full` we make sure to capture all the sections in Nsight Compute profiler):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile the selected kernel in the solution with Nsight compute\n",
    "!cd ../../source_code/openacc && ncu --set full --kernel-regex pair_gpu_182_gpu --launch-skip 1 --launch-count 1 -o rdf_collapse_solution ./rdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's checkout the Nsight Compute profiler's report together. [Download the profiler output](../../source_code/openacc/rdf_collapse_solution.ncu-rep) and open it via the GUI. Let's checkout the first section called \"GPU Speed Of Light\". This section gives an overview of the utilization for compute and memory resources on the GPU. As you can see from the below screenshot, the Speed of Light (SOL) reports the achieved percentage of utilization of 30.58% for SM and 19.47% for memory. \n",
    "\n",
    "\n",
    "<img src=\"../images/sol.png\">\n",
    "\n",
    "\n",
    "**Extra**: If you can use the baseline feature on the Nsight Compute and compare the analysis of the kernel from this version of the RDF (which uses data directives and collapse clause) with the very first parallel version where we only added parallel directives and used managed memory, you can see how much improvements we got (see the below screenshot for reference):\n",
    "\n",
    "<img src=\"../images/sol_baseline.png\">\n",
    "\n",
    "It is clear that we were able to reduce the execution time to half(red rectangle) and increase the SM and memory utilization (green rectangle). However, as you see the device is still underutilized. Let's look at the roofline analysis which indicates that the application is latency bound and the kernel exhibits low compute throughput and memory bandwidth utilization relative to the peak performance of the device this was profiled on.\n",
    "\n",
    "<img src=\"../images/roofline_collapse.png\">\n",
    "\n",
    "The Nsight Compute profiler suggests us to checkout the \"Scheduler Statistics\" and \"Warp State Statistics\" sections to see how we can improve the utilization. By checking the the \"Scheduler Statistics\" section, we can see that for this kernel the scheduler does not issue an instruction every cycle. We have many active warps (average of 11.29 out of 16) but only an average of 0.68 warps are eligible per cycle. This results in no instruction being issues and the issue slot remains unused. The \"Warp State Statistics\" section can indicate which stall reasons causes this.\n",
    "\n",
    "<img src=\"../images/scheduler_collapse.png\">\n",
    "\n",
    "From the  \"Warp State Statistics\" section, we can see that the most important stall reason is LG (local/global) Throttle due to extremely frequent memory instructions.\n",
    "\n",
    "<img src=\"../images/warp_collapse.png\">\n",
    "\n",
    "Now, let's have a look at the \"Source Counters\" section located at the end of \"Details\" page of the profiler report. \n",
    "\n",
    "<img src=\"../images/source_loc.png\">\n",
    "\n",
    "\n",
    "The section contains tables indicating the N highest or lowest values of one or more metrics in the selected kernel source code. Hotspot tables point out performance problems in the source. \n",
    "\n",
    "<img src=\"../images/source_collapse.png\">\n",
    "\n",
    "We can select the location links to navigate directly to this location in the \"Source\" page. Moreover, you can hover the mouse over a value to see which metrics contribute to it.\n",
    "\n",
    "<img src=\"../images/source_hover.png\">\n",
    "\n",
    "The \"Source\" page displays metrics that can be correlated with source code. It is filtered to only show (SASS) functions that were executed in the kernel launch.\n",
    "\n",
    "<!--<img src=\"../images/source_sass_collapse.png\">-->\n",
    "\n",
    "<img src=\"../images/source_sass.png\">\n",
    "\n",
    "The \"Source\" section in the \"Details\" page indicates  that the issue is *uncoalesced Global memory access*. \n",
    "\n",
    "<img src=\"../images/uncoalesced_hint.png\">\n",
    "\n",
    "**Memory Coalescing**\n",
    "\n",
    "On GPUs, threads are executed in warps. When we have a group of 32 contiguous threads called *warp* accessing adjacent locations in memory, we have *Coalesced memory* access and as a result we have few transactions and higher utilization. However, if a warp of 32 threads accessing scattered memory locations, then we have *Uncoalesced  memory* access and this results in high number of transactions and low utilization.\n",
    "\n",
    "\n",
    "<img src=\"../images/coalesced_mem.png\">\n",
    "\n",
    "Without changing the data structure and refactoring the code, we cannot fix this issue and improve the performance further using OpenACC in a straightforward easier way. The next step would be to look into how to optimize this application further with CUDA and perhaps take advantage of shared memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-Lab Summary\n",
    "\n",
    "If you would like to download this lab for later viewing, it is recommend you go to your browsers File menu (not the Jupyter notebook file menu) and save the complete web page.  This will ensure the images are copied down as well. You can also execute the following cell block to create a zip-file of the files you've been working on, and download it with the link below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ..\n",
    "rm -f nways_files.zip\n",
    "zip -r nways_files.zip *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After** executing the above zip command, you should be able to download the zip file [here](../nways_files.zip). Let us now go back to parallelizing our code using other approaches.\n",
    "\n",
    "**IMPORTANT**: Please click on **HOME** to go back to the main notebook for *N ways of GPU programming for MD* code.\n",
    "\n",
    "-----\n",
    "\n",
    "# <p style=\"text-align:center;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\"> <a href=../../../nways_MD_start.ipynb>HOME</a></p>\n",
    "\n",
    "-----\n",
    "\n",
    "\n",
    "# Links and Resources\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "[NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute)\n",
    "\n",
    "[CUDA Toolkit Download](https://developer.nvidia.com/cuda-downloads)\n",
    "\n",
    "**NOTE**: To be able to see the Nsight System profiler output, please download Nsight System latest version from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [OpenACC Resources](https://www.openacc.org/resources) and join our [OpenACC Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- \n",
    "\n",
    "## Licensing \n",
    "\n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
