{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Remarks\n",
    "\n",
    "In this tutorial we took an approach where same algorithm was ported to GPU using different popular methods. Each method has it strengths and suffices a purpose for which it was created. From a developer point of view below listed are some key parameters which are crucial to any development exercise: \n",
    "\n",
    "- **Ease of Programming**: How much in-depth knowledge of processor architecture is required for a developer before starting to convert the code to GPU?\n",
    "- **Performance**: How much effort is required to reach desirable performance on a particular architecture.\n",
    "- **Portability**: To what extent does the same code run on multiple architecture? What provisions are provided by programming approach to target different platforms?\n",
    "- **Support**: The overall ecosystem and support by the community.\n",
    "    - Which all compilers implement the standard?\n",
    "    - Which all languages are supported?\n",
    "    - Which all applications make use it?\n",
    "    - How easy or difficult it is to profile/debug the application?\n",
    "    \n",
    "Let us try to create a high level buckets for each of these parameter above with a limited scope of GPU support:\n",
    "\n",
    "| | |  |  | \n",
    "| :--- | :--- | :--- | :--- |\n",
    "| Ease of Programming | Low: Minimal architecture specific knowledge needed  | Intermediate: Mimimal changes expected in code design.  Using these along with architecture knowledge helps in better performance | High: In-Depth GPU architecture knowledge must |\n",
    "| Performance  | Depends: Based on the complexity/type of application the performance may vary | High: Exposes methods to get good performance. These methods are integral part of design and exposed to programmer at various granularities | Best: Full control to developers to control parallelism and memory access |\n",
    "| Portability | Integral: Part of the key objective  | Limited: Works only on specific platform | | \n",
    "| Support | Established: Proven over years and support by multiple vendors for GPU | Emerging: Gaining traction by multiple vendors for GPU  | |\n",
    "\n",
    "There is a very thin line between these categories and within that limited scope and view we could categorize different approaches as follows:\n",
    "\n",
    " \n",
    "| | OpenACC | OpenMP | stdpar | Kokkos | CUDA Laguages |\n",
    "| --- | --- | --- | --- | --- | --- |\n",
    "| Ease | High  | High | High  | Intermediate | Low |\n",
    "| Performance  | Depends | Depends | Depends | High | Best |\n",
    "| Portability | Integral  | Integral | Integral | Integral | Limited |\n",
    "| Support | Established | Emerging | Emerging | Established | Established |\n",
    "\n",
    "Below given are points that will help users as there is no one programming model that fits all needs.\n",
    "\n",
    "## Ease of Programming\n",
    "- The directive‐based OpenMP and OpenACC programming models are generally least intrusive when applied to the loops. \n",
    "- Kokkos required restructuring of the existing code for the parallel dispatch via functors or lambda functions\n",
    "- CUDA required a comparable amount of rewriting effort, in particular, to map the loops onto a CUDA grid of threads and thread blocks\n",
    "- stdpar also required us to change the constructs to make use of C++17 templates and may be preferred for new developments having C++ template style coding. \n",
    "- The overhead for OpenMP and OpenACC in terms of lines of code is the smallest, followed by stdpar and Kokkos\n",
    "\n",
    "## Performance\n",
    "While we have not gone into the details of optimization for any of these programming model the analysis provided here is based on the general design of the programming model itself.\n",
    "\n",
    "- Kokkos when compiled enables the use of correct compiler optimization flags for the respective platform, while for the other frameworks, the user has to set these flags manually. This gives kokkos an upper hand over OpenACC and OpenMP. \n",
    "- OpenACC and OpenMP abstract model defines a least common denominator for accelerator devices, but cannot represent architectural specifics of these devices without making the language less portable.\n",
    "- stdpar on the other hand is more abstract and gives less control to developers to optimize the code\n",
    "\n",
    "## Portability\n",
    "We observed the same code being run on moth multicore and GPU using OpenMP, OpenACC, Kokkos and stdpar. The point we highlight here is how a programming model supports the divergent cases where developers may choose to use different directive variant to get more performance. In a real application the tolerance for this portability/performance trade-off will vary according to the needs of the programmer and application \n",
    "- OpenMP supports [Metadirective](https://www.openmp.org/spec-html/5.0/openmpsu28.html) where the developer can choose to activate different directive variant based on the condition selected.\n",
    "- In OpenACC when using ```kernel``` construct, the compiler is responsible for mapping and partitioning the program to the underlying hardware. Since the compiler will mostly take care of the parallelization issues, the descriptive approach may generate performance code for specific architecture. The downside is the quality of the generated accelerated code depends significantly on the capability of the compiler used and hence the term \"may\".\n",
    "\n",
    "\n",
    "## Support\n",
    "- Kokkos project is very well documented and the developers support on GitHub is excellent \n",
    "- OpenACC implementation is present in most popular compilers like NVIDIA HPC SDK, PGI, GCC, Clang and CRAY. \n",
    "- OpenMP GPU support is currently available on limited compilers but being the most supported programming model for multicore it is matter of time when it comes at par with other models for GPU support.\n",
    "- stdpar being part of the C++ standard is bound to become integral part of most compiler supporting parallelism. \n",
    "\n",
    "\n",
    "Parallel Computing in general has been a difficult task and requires developers not just to know a programming approach but also think in parallel. While this tutorial provide you a good start, it is highly recommended to go through Profiling and Optimization bootcamps as next steps.\n",
    "\n",
    "-----\n",
    "\n",
    "# <div style=\"text-align: center ;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\">[HOME](../../nways_MD_start.ipynb)</div>\n",
    "\n",
    "-----\n",
    "\n",
    "# Links and Resources\n",
    "[OpenACC API guide](https://www.openacc.org/sites/default/files/inline-files/OpenACC%20API%202.6%20Reference%20Guide.pdf)\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "[NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute)\n",
    "\n",
    "[CUDA Toolkit Download](https://developer.nvidia.com/cuda-downloads)\n",
    "\n",
    "**NOTE**: To be able to see the Nsight System profiler output, please download Nsight System latest version from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [OpenACC Resources](https://www.openacc.org/resources) and join our [OpenACC Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- \n",
    "\n",
    "## Licensing \n",
    "\n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
